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i«.  Abstract  jf,is  report  is  one  of  a  series  of  documented  studies  directed  to  the  im¬ 
provement  and  standardization  of  aircraft  alerting  systems.  The  purpose  of  the  study 
was  to  develop  and  validate,  through  simulation,  functional  design  criteria  that  can  be 
used  in  designing  effective  aircraft  alerting  systems.  The  major  objectives  of  this 
phase  of  the  study  were  to: 

Resolve  system  component  questions' 

Validate  the  two  candidate  system -concepts  by  comparison  to  a  representative 
baseline  system,  for  both  the  pilot  and  flight  engineer  stations! 

Evaluate  presentation  media  and  display  formats  for  time-criticSi  warnings/ 

Jtevelop  guidelines  for  the  design  of  alerting  systems. 

The  system  component  questions  were  resolved  in  a  series  of  experiments  and  the  results 
were  integrated  into  two  system  concepts  for  validation.  In  the  validation  tests,  line 
qualified  pilots  exercised  the  three  alerting  systems  in  a  full-mission,  fixed-base 
simulator.  In  all  areas  of  measurement  pilot  performance  was  as  good  or  better  with 
the  two  candidate  system  designs  than  with  the  conventional,  baseline  system.  The 
results  indicate  that  a  flight  engineer  station  should  contain  the  visual  display  com¬ 
ponents  offered  by  the  candidate  system  concepts.  Voice  was  the  recommended  medium  of 
presentation  when  rapid  response  is  required.  The  time-critical  display  tests  showed 
that  fastest  pilot  response  performance  was  obtained  when  graphic  guidance  alerts  were 
placed  in  the  pilot's  primary  field  of  view.  The  functional  design  criteria  that  were 
developed  are  contained  in  "Aircraft  Alerting  System  Design  Guidelines,"  Report  Number 
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This  report  is  the  result  of  several  years  of  research  sponsored  by  the  FAA 
directed  toward  the  improvement  and  standardization  of  aircraft  alerting 
systems.  This  present  study  was  conducted  as  a  joint  effort  by  the  three 
major  U.S.A.  manufacturers  of  commercial  transport  aircraft:  Boeing, 
Lockheed,  and  McDonnell  Douglas.  The  primary  purpose  of  this  volume  is  to 
report  the  system  validation  and  the  time-critical  presentation  media  tests. 
The  objectives  of  these  tests  were  to  validate  the  system  design  concepts 
identified  in  the  first  phase  of  the  study  and  to  evaluate  presentation 
methods  for  alerts  which  announce  situations  or  conditions  which  require  an 
extremely  rapid  solution. 
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1.0  INTRODUCTION 


This  contract  Is  the  fourth  In  a  series  directed  toward  the  Improvement  and 
standardization  of  aircraft  alerting  systems.  The  effort  began  In  1973  with  a 
study  of  concepts  for  an  Independent  altitude  monitor,  and  has  evolved  Into 
the  study  of  advanced  cockpit  alerting  system  concepts  and  design  criteria. 
With  each  effort  It  became  Increasingly  obvious  that  the  design  of  aircraft 
alerting  systems  has  not  followed  a  systematic  approach,  but  rather  has  been 
detailed  by  the  requirements  of  Individual  systems.  As  requirements  for  new 
alerts  were  Identified  these  additional  alerting  elements  were  placed  In  the 
flight  deck  with  little  regard  to  Integrating  them  with  other  alerting 
elements.  While  a  good  data  base  of  philosophies,  pilot-response 
characteristics,  and  design  guidelines  were  obtained.  Increasing  evidence  of 
alert  proliferation  and  Inconsistent  application  of  alerting  concepts  were 
found  (Cooper,  1977;  and  Veltengruber,  Boucek,  and  Smith,  1977).  Those 
studies  clearly  Indicated  that  the  aircraft  manufacturing  Industry  needs  a  set 
of  guidelines  for  designing  aircraft  alerting  systems  for  the  next  generation 
of  commercial  transports.  Those  studies  also  provided  general  recommendations 
for  standardizing  alerting  functions  and  methods.  However,  those  studies  also 
assessed  the  adequacy  of  the  existing  alert  system  data  base  and  recommended 
additional  comparative  testing  of  not  only  alerting  system  elements  but  also 
full  alerting  system  concepts  be  used  to  refine  and  validate  the  design 
guidelines.  It  was  recognized  that  any  recommendations  must  be  based  on  data 
and  reflect  a  consensus  of  manufacturers,  certificating  organization, 
operators  and  pilots  to  be  of  value. 

The  present  contract  was  awarded  to  obtain  empirical  pilot  performance  data  to 
refine  and  validate  the  preliminary  alerting  system  concepts  and  criteria. 

The  contract  was  performed  In  three  phases.  The  first  phase  consisted  of  an 
evaluation  of  Individual  alerting  system  components  (e.g.,  master  visual 
alert,  voice  Information  display),  and  resulted  In  the  definition  of  candidate 
alerting  concepts.  The  assinptlons  upon  which  the  first  stuty  was  based  can 
be  seen  In  Table  1.0-1.  The  candidate  alerting  concepts  were  developed  to 
cover  the  alerting  requirements  for  a  three  level  alerting  system  (warning, 
caution  and  advisory).  Table  1.0-2  defines  these  levels  and  Illustrates  the 
alerting  system  components  required  to  mechanize  them.  The  Phase  1  effort  Is 


Table  1.0-1.  System  Assumptions  for  Aircraft  Alerting  Systemt  Standerdiza  tion  Study 


•  No  nonverbal  aural  alerting  tyetern  with  different  alerts  for  each  condition  or  alert  (large  number  of  aurals) 

•  No  incandescent  light  or  fixed- legend  display  for  primary  central  display  unit 

•  Secondary  subsystem  indicators  will  be  reflected  on  the  central  display 

•  Dual-channel  auditory  and  visual  presentation  for  some  if  not  all  alerts 

•  Primary  visual  system  will  be  programmable;  subsystem  indication  may  be  fixed 

•  Auditory  system  voice  components 

•  Auditory  system  tone  components 

•  System  direction  toward  an  electronic  flight  deck 

•  Form  of  prioritization  implemented 

•  Form  of  automated  Inhibition  needed;  e.g.,  don't  use  voice  when  it  might  conflict  with  ATC  communicetions 

•  Computing  capability  (smart  system)  to  handle  prioritization,  inhibit,  and  other  system  logic 

•  Design  for  the  quiet.dark  cockpit 

•  May  want  some  alerts  to  bypass  computer  for  backup  in  a  failure  mode 

•  Central  display  primarily  alphanumeric  but  may  have  graphic  or  symbolic  capabilities 

•  Automatic  indication  clearing  when  fault  or  alert  condition  no  longer  exists 

•  Best  available  speech -generation  equipment 

•  System  based  on  four  condition  levels;  i.e„  ARP450D. 

•  Warning 

•  Caution 

•  Advisory 

•  Information 

•  Central  display  with  color  capability 

•  Capability  of  readily  accommodating  all  present  and  future  alerting  functions;  e.g.,  BCAS,  GPWS 

•  Basic  functions  to  indude- 

•  Alert  (attention  getting) 

•  Inform  (identify  the  problem) 

•  Guide  crew  action 

•  Provide  feedback 

•  Includes  interactive  capabilities 


documented  In  Boucek,  Erickson,  Berson,  Hanson,  Leffler,  and  Po-Chedley 
(1980).  The  second  phase  consisted  of  developing  a  detailed  test  plan  for 
evaluating  the  candidate  concepts  in  a  full-mission  simulation.  In  the  third 
phase,  line-qualified  pilots  exercised  the  resulting  alerting  systems  In  a 
simulator,  and  a  set  of  design  guidelines  directed  at  the  Improvement  and 
standardl zatlon  of  advanced  aircraft  alerting  systems  was  developed.  The 
Intent  of  these  guidelines  Is  not  to  define  a  single  alerting  system  design 
that  each  manufacturer  must  use,  but  rather  to  provide  functional  design 
criteria  that  can  be  used  to  develop  effective  alerting  systems,  and  to 
promote  standardization  within  the  Industry.  These  guidelines,  provided  In 


a 


f 


i 

I 

I 


Table  1.0-2.  Alerting  System  Categorization 


Condition 

Critaria 

Alert  system  characteristics 

Visual 

Aural 

Tactile 

Warning 

Emergency  oparational  or  aircraft 

Master  visual  (red) 

Unique 

Stick 

systam  conditions  that  require 

plus  centrally  located 

attention- 

shaker 

immediate  corrective  or  comoensatorv 

alphanumeric 

getting 

(if 

crew  action 

readout  (rod) 

warning 
sound 
plus  voice* 

required) 

Caution 

Abnormal  operational  or  aircraft 
system  conditions  that  require 
immediate  crew  awareness  and  require 
prompt  corrective  or  compensatory 
crew  action 

Master  visual  (amber) 

plus  centrally  located 

alphanumeric 

readout 

(amber) 

Unique 

attention- 

getting 

caution 

sound 

plus  voice* 

None 

Advisory 

Operational  or  aircraft  system  condi¬ 
tions  that  reauire  crew  awareness 
and  may  require  crew  action 

Centrally  located 
alphanumeric 
readout 
(unique  color) 

Unique 

attention- 

getting 

advisory 

sound 

None 

Information 

Operational  or  aircraft  system 
conditions  that  require  cockpit 
indications,  but  not  necessarily  as 
part  of  the  integrated  warning  system 

Discrete  indication 
(green  and  white) 

None 

None 

‘Voice  is  pilot  selectable. 


Volume  II  of  this  report  (Berson,  Po-Chedley,  Boucek,  Hanson,  Leffler,  and 
Wasson  1981),  have  been  substantiated  by  experimental  data  and  do  reflect 
views  of  commercial  transport  aircraft  manufacturers,  certification 
authorities,  airline  operators  and  pilots. 

The  objectives  of  these  guidelines  are  to  provide  standardization  in  alerting 
system  design  and  methods  to  reduce  the  overall  number  of  discrete  visual  and 
aural  alerts  In  the  next  generation  of  commercial  jet  transports.  By 
accomplishing  these  objectives  the  demands  on  crew  Information  processing  and 
memory  capabilities  should  be  reduced.  The  time  required  to  detect  and  assess 
failure  conditions  should  be  minimized,  as  well  as  the  time  to  Initiate  the 
appropriate  corrective  actions.  There  should  be  fewer  distractions  from  the 
crew  tasks  of  aircraft  control  and  communications.  By  standardizing  the 
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alerting  system,  all  airframe  manufacturers,  airline  operators  and  pilots  trill 
share  the  benefits  of  a  commonality  of  design.  The  standardized  system  should 
also  provide  for  alerting  system  growth  and  Improvement  In  a  form  that  does 
not  necessitate  additional  discrete  annunciators. 

The  present  study  was  conducted  as  a  joint  effort  between  Boeing,  McDonnell 
Douglas  and  Lockheed  Aircraft  Companies,  and  was  sponsored  by  the  Federal 
Aviation  Administration. 

This  report  contains  two  volumes.  Volume  1  describes  the: 

•  Supplemental  tests  that  were  conducted  to  select  between  alternative 
alerting  system  elements. 

•  Simulation  tests  that  were  conducted  on  the  candidate  alerting  concepts. 
Identified  In  Pba'e  1,  to  validate  these  concepts  by  comparison  to  a 
conventional  "baseline"  system,  {l.e.,  representative  of  systems 
currently  In  u%«>.  validation  tests  were  conducted  for  pilot  and  flight 
engineer  stations. 

•  Simulation  tests  that  evaluated  alternative  concepts  for  providing  time- 
critical  warning  Information  to  the  flight. 

The  second  volime  contains  the  alerting  system  design  guidelines. 

1.1  REPORT  ORGANIZATION 

Section  2  of  this  report  contains  an  executive  summary  of  the  major  activities 
and  findings  of  the  Phase  3  testing  effort.  The  supplemental  studies 
conducted  to  select  between  alternative  concepts  for  Implementing  the  alerting 
functions,  and  the  results  of  these  studies  are  described  In  Section  3. 

Section  4  describes  the  system  validation  tests,  Including  the  tests  conducted 
at  the  pilot  and  flight  engineers  stations,  and  the  time-critical  tests. 
Discussions  of  the  major  findings  and  the  conclusions  drawn  from  this  data  are 
presented  In  Section  5,  and  Section  6  describes  additional  study  efforts  that 
should  be  undertaken  to  Incorporate  aircraft  or  operational  specific 
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considerations  Into  the  alerting  systea  design,  and  then  to  lapleaent  the 
design  In  hardware  for  a  flight  evaluation  phase. 

The  Appendices  at  the  end  of  Mis  report  describe  the  facilities  used  at 
Boeing  and  Douglas  to  perfora  the  systea  validation  and  suppleaental  tests  and 
contain  exaaple  Illustrations  of  the  Interactive  capabilities  of  the  visual 
Inforaatlon  display.  Also  Included  are  the  questionnaires  that  were  used  to 
obtain  pilot  Input  for  Incorporation  Into  the  guidelines. 


2.0  EXECUTIVE  SUMMARY 


2.1  PROGRAM  BACKGROUND 

The  present  contract,  performed  In  three  phases,  was  directed  toward  the 
Improvement  and  standardization  of  aircraft  alerting  systems.  In  Phase  1  the 
primary  functions  of  an  advanced  alerting  system  were  defined,  and  alternative 
concepts  for  Implementing  those  functions  were  evaluated  through  simulation. 
The  final  effort  of  Phase  1  was  to  develop  candidate  alerting  concepts.  These 
concepts  were  to  be  Implemented  by  combining  the  Individual  functional 
elements,  evaluated  In  Phase  1,  Into  systems  that  best  satisfied  the 
requirements  for  crew  alerting.  The  systems  were  composed  of  elements  forming 
four  major  system  components,  master  visual  alerts,  master  aural  alerts, 
visual  Information  display,  and  the  voice  Information  display.  Two  systems 
were  defined.  The  two  candidate  designs,  presented  In  Tables  2.1-1  through 
2.1-4,  are  quite  similar,  differing  only  in  their  implementation  of  the  voice 
Information  display.  System  A  provides  automatic  voice  messages  for  all 
warnings,  whereas.  System  B  provides  voice  messages  for  warnings  and  cautions 
but  which  are  annunciated  only  when  selected  by  the  pilot.  To  provide  a 
baseline  for  validating  the  advanced  systems,  a  conventional  system, 
representatl ve  of  alerting  schemes  currently  In  use,  was  developed.  The 
characteristics  of  the  conventional  system  are  presented  In  Table  2.1-5. 

In  the  process  of  Identifying  the  characteristics  of  the  candidate  systems  a 
number  of  questions  arose  concerning  alternative  methods  for  accomplishing 
alerting  system  functions.  Alternative  designs  were  Identified  for 
accomplishing  the  master  visual  alert,  for  performing  the  alerting  and 
Informing  functions  for  advisories,  and  for  annunciating  multi pie- alerting 
situations.  It  was  also  recognized  that  some  conditions  or  situations  may 
require  crew  action  to  be  accomplished  In  an  extremely  short  period  of  time. 
These  situations  were  termed  "time-critical",  and  alternative  presentation 
media  and  formats  were  developed  for  evaluation  In  Phase  3. 
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Variable 


A 


ConoaptB 


Number 

Two 

• 

Location 

Near  Ifrdeg  cones 

• 

Flaah 

No 

• 

Brightness 

<6  to  ISO  fL 

• 

Size 

Ideg 

« 

Cancellation  logic 

Manual  and  automatic 

• 

Duty  cycle 

N/A 

• 

Color 

Rad,  yellow 

• 

Table  2. 1-2.  Aura!  Matter  Alert 


Variable 

Concept  A 

Concept  B 

Number 

Three 

• 

Signal-to-noise  ratio 

6  to  tO  dB 

• 

Cancellation 

Manual  and  automatic 

• 

Stereo  type  alerts 

No 

• 

Duty  cycle 

N/A 

• 

Spectral  character 

In  guidelines 

• 

Location 

90  dag 

• 

Madcing 

Controlled  via  design 

• 

'Variable  is  identical  for  both  concepts 


Table  2. 1-3.  Visual  Information  Display 


Variable 

Concept  A 

Concept  B 

Location 

Central  display 

• 

Format 

Priority  and  chronological 

• 

Overflow 

Paging 

• 

Store-recall 

Yes,  except  for  warnings 

• 

Brightness 

15  to  150  fL 

• 

Cues  and  aids 

Box,  arrow,  etc. 

• 

Content 

Short  phrase  (syntax) 

• 

Character  size 

14  min 

• 

Character  spacing 

7  min 

• 

Legibility 

In  literature 

» 

Table  2. 1-4.  Voice  Information  Display 


Variable 

Concept  A 

Concept  B 

Type 

Warnings 

Warnings  and  caution  elective 

Format 

Phrase 

* 

Model  (M/F) 

Female 

« 

Inflection 

Monotone 

• 

Masking 

Controlled  by  design 

• 

Repetition 

Yes 

• 

Cancellation 

Manual 

Manual  switch 

Content 

Status 

• 

Signal-to-noise  ratio 

5  to  10  dB 

• 

Multiple  alerts 

In  sequence  with 
repetition 

No 

Store-recall 

No 

Yes 

Spectral  character 

Guidelines 

• 

Location 

90  deg 

• 

Variable  it  identical  for  both  concepts. 


Table  2. 1-5.  Baseline  Configuration 


Yes/no 

Location 

Color/tone 

Flash 

Master  warning 

Yes 

Glareshield 

Red/no 

No 

Single  master 

Split  legend 

Master  caution 

Yes 

Glareshield 

Amber 

No 

Central  display 

Fixed  legend 

Yes 

Central  panel 

Red,  amber, 
blue 

Yes,  without 
master 

Alphanumeric 

Yes 

Red,  amber, 
blue 

No,  with 
master 

Monochrome 

No 

Distributed  annunciators 

Yes 

N/A 

Red,  amber, 
blue 

No 

ADDITIONAL  FEATURES. 


Dedicated  tones: 

Electronic-mechanical  mix;  also  includes  SELCAL,  CREW  CALL; 

cancellable,  dedicated  cancel  switches 

Voice  message: 

GROUND  PROXIMITY,  cockpit  speaker  environment 

Inhibits: 

No 

Cancel/recall: 

Yes;  can  cancel  masters  and  central  display;  cannot  cancel  distributed 

annunciators 
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In  Phase  2  detailed  test  plans  were  developed  for: 

•  Resolving  the  system  component  questions. 

•  Validating  the  two  candidate  system  concepts  by  comparison  to  a 
representatl ve  baseline  system,  for  both  the  pilot  and  flight  engineer 
stations. 

•  Evaluating  presentation  media  and  display  formats  for  time-critical 
warnings. 

The  major  objectives  of  Phase  3  were  to  perform  the  evaluations  discussed 
above,  to  analyze  the  resulting  data  with  all  other  available  data,  and  to 
develop  a  set  of  design  guidelines  for  alerting  systems.  The  intent  of  the 
guidelines  is  not  to  define  a  single  design  that  all  airframe  manufacturers 
must  use,  but  rather  to  identify  functional  design  criteria  that  can  be  used 
to  develop  effective  alerting  systems,  and  to  promote  standardization  within 
the  industry.  This  volume  describes  the  Phase  3  testing  effort,  and  Volume  2 
contains  the  design  guidelines. 

2.2  SUPPLEMENTAL  TESTS 

In  developing  the  candidate  system  concepts  in  Phase  1,  a  numbe*  of  quest* 
arose  concerning  the  design  of  various  alerting  system  elements.  Additional 
tests  were  designed  to  obtain  resolution  to  those  questions.  These 
supplemental  tests  were  conducted  at  McDonnell  Douglas,  Long  Beach,  California 
and  were  completed  prior  to  the  major  Phase  3,  simulation  testing,  which  was 
conducted  by  Boeing  Commercial  Airplane  Company,  Seattle,  Washington.  The 
areas  that  required  resolution  prior  to  conducting  the  Phase  3  testing  are 
summarized  in  the  following  paragraphs  and  are  described  in  detail  in  Section 
4.  The  supplemental  tests  were  divided  into  issues  that  addressed  system 
component  design,  and  alerting  system  logic.  The  vast  majority  of  data  used 
to  resolve  these  issues  were  obtained  from  questionnaire  surveys  of  25  current 
line-qualified  pilots,  following  demonstrations  of  alternative  designs  in  the 
Douglas  simulator. 


2.2.1  SYSTEM  COMPONENT  TESTS 


2.2.1.1  Design  of  the  Master  Visual  Alert  -  This  test  Involved  evaluating  the 
relative  advantages  and  disadvantages  of  separate  master  visual  lights  for 
warnings  and  cautions  versus  a  single  split-legend  Indicator  labeled  WARNING 
on  top  and  CAUTION  on  the  bottom.  The  results  of  this  test  indicated  that  the 
pilots  were  more  concerned  with  the  size  and  location  of  the  master  visual 
alerts  than  they  were  with  whether  they  were  separate  or  combined. 

2.2. 1.2  Requirement  for  a  Master  Visual  Advisory  Alert  -  This  test  addressed 
whether  a  master  advisory  alert  should  be  provided  in  alerting  systems.  The 
results  of  the  test  Indicated  that  a  master  visual  alert  was  preferred  in 
conjunction  with  the  annunciation  of  advisory  alerts  via  a  master  aural  alert 
(single  stroke  chime),  and  presenting  the  alert  on  a  visual  information 
display. 

2.2.1.3  Interactive  Functions  -  This  evaluation  addressed  the  mechanisms  to 
enable  the  crew  to  Interact  with  the  alerting  system.  One  question  was  the 
desirability  of  providing  the  crew  with  the  capability  to  store  alerts  (l.e., 
remove  them  from  the  visual  display  and  store  them  In  memory),  and  to  recall 
them  when  desired.  Total  store/ recall  (all  alerts  at  one  time)  and  selective 
store/recall  (one  at  a  time)  capabilities  were  investigated.  The  study 
results  Indicated  that  a  combination- of  total  and  selective  store/recall 
should  be  provided. 

A  second  Issue  was  that  of  providing  automatic  checkl 1st/ procedural 
Information  to  aid  the  crew  In  responding  to  emergency  situations.  The  vast 
majority  of  pilots  surveyed  (23  out  of  25)  Indicated  that  this  capability 
should  be  Included  In  conjunction  with  aircraft  alerting  systems. 

The  pilots  were  also  questioned  about  the  desirability  of 
activating/cancelling  aural  alerts  and  messages.  The  results  showed  a  clear 
preference  for  manual  cancellation. 

2.2.14  Verbal  Versus  Sound  Master  Alerts  -  This  test  was  a  subjective 
comparison  of  using  tones  or  verbal  alerts  (presenting  the  word  "WARNING", 
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"CAUTION1*,  or  "ADVISORY")  as  the  aural  attention-getting  mechanism.  Slightly 
more  than  50  percent  of  the  pilots  Interviewed  favored  the  verbal  alerts  over 
the  discrete  sounds.  However,  since  the  preference  was  not  statistically 
significant  and  there  Is  a  greater  probability  of  the  verbal  alerts 
Interfering  with  other  flight  deck  communications  (e.g.,  ATC,  crew 
conversations),  tones  were  selected  for  Implementation  In  the  candidate 
systems  for  the  Phase  3  evaluation. 

2.2.2  SYSTEM  LOGIC  TESTS 

2.22.1  Prioritization  -  This  test  addressed  the  Issue  of  prioritizing  alerts 
within  the  three  alerting  categories  (l.e.,  warning,  caution,  and  advisory)  on 
the  visual  Information  display.  Optimally,  the  most  Important  message  would 
always  be  located  at  the  top  of  the  display,  with  lower  priority  alerts  below. 
The  results  of  this  survey  Indicated  that  while  alert  prioritization  would 
facilitate  crew  performance,  there  was  very  little  agreement  on  how 
prioritization  should  be  accomplished. 

22.2.2  Inhibit  Logic  -  The  Issue  of  inhibiting  low  level  alerts  during  high 
workload  flight  phases  was  investigated.  As  with  prioritization,  most  pilots 
indicated  that  Inhibit  logic  should  be  Incorporated  in  the  alerting  system, 
however  little  agreement  was  obtained  on  what  components  to  Inhibit,  or  when 
to  do  so. 

2.22.3  Visual  Message  Syntax  -  The  objective  of  this  study  was  to  survey  the 
major  aircraft  manufacturers  to  identify  the  most  prevalent  alert  format.  The 
results  of  this  survey  indicated  that  while  no  standard  format  existed,  most 
alerts  were  presented  with  the  nature  of  the  problem  preceeding  the  location 
of  the  problem.  The  majority  of  the  25  pilots  (80%)  also  preferred  the  format 
which  had  the  general  heading,  followed  by  the  specific  subsystem/location, 
and  the  nature  of  the  problem  (e.g.,  ENGINE  NUMBER  1  FIRE).  However,  they 
stated  that  a  standard  syntax  may  not  be  appropriate  for  all  alerts,  and  that 
while  a  standard  message  syntax  Is  desirable,  It  should  be  subordinate  to  a 
clear  statement  of  the  problem. 


2.2.2A  Multiple  Verbal  Alerts  -  This  test  Investigated  the  sequencing  of 
multiple  verbal  alerts.  The  alternatives  Investigated  Included:  (1) 
Prioritizing  the  alert  messages  and  annunciating  the  highest  priority  alert; 
(2)  Annunciate  the  highest  priority  alert  for  a  fixed  number  of  repetitions, 
then  cancel  It,  and  present  the  next  highest  message,  and  so  on;  and  (3) 
Annunciate  the  message  "MULTIPLE  ALERTS"  to  direct  the  crews*  attention  to  the 
visual  Information  display  for  the  specific  fault  messages.  The  pilots  showed 
a  preference  for  the  annunciation  of  the  message  "MULTIPLE  ALERTS". 

The  results  obtained  via  these  supplemental  tests  were  Incorporated  Into  the 
Phase  3  validation  tests  described  In  the  following  paragraphs. 

2.3  SYSTEM  VALIDATION  TESTS 

The  two  candidate  advanced  systems  and  the  conventional  alerting  system, 
defined  In  Phase  1,  were  Implemented  In  the  Visual  Flight  Simulation  Facility 
at  Boeing.  Fourteen  line-qualified  pilots,  with  an  average  of  13,600  hours 
flight  experience,  participated  In  the  tests.  Each  flew  eighteen  test  flights 
of  31  minutes  In  length  and  responded  to  162  alerts  of  var1ous~urgency  levels. 
Six  of  the  flights  were  designed  validating  the  advanced  systems.  Half  the 
pilots  flew  System  A  while  the  other  half  flew  System  B.  All  pilots  flew  two 
trials  with  the  conventional  system. 

To  simulate  a  flight  deck  environment  and  work  pattern,  a  realistic  aircraft 
model  was  used  for  the  basic  flying  task.  In  addition,  the  pilots  were 
required  to  fly  a  prescribed  flight  plan  (takeoff,  climb,  cruise,  descent  and 
land),  respond  to  ATC  directives,  locate  and  report  targets  In  the  external 
visual  scene,  and  respond  to  alerts. 

In  the  Flight  Engineer  tests,  conducted  simultaneously  with  the  pilot  tests, 
the  test  subjects  were  provided  a  workload  to  simulate  worst-case  or  near 
worst-case  conditions.  The  Flight  Engineers'  tasks  Included  reading 
Instrunents,  logging  problems  or  faults  and  their  time  of  occurrence,  and 
locating  targets  In  the  external  visual  scene.  Each  pilot  flew  four  Flight 
Engineer  trials,  two  with  a  conventional  system,  and  two  with  an  advanced 
system.  The  results  of  the  Pilot  and  Flight  Engineer  validation  tests  are 
described  In  detail  In  Section  S,  and  are  simmarlzed  below: 


2.3.1  PILOT  STATION  TESTS 


Using  the  advanced  alerting  systems,  pilot  performance  was  as  good  as  or 
better  than  that  obtained  with  the  conventional  alerting  system,  no  matter 
what  performance  measure  was  used.  Pilot  response  and  detection  times  were 
significantly  shorter,  and  fewer  alerts  were  missed  with  Systems  A  and  B,  than 
with  the  conventional  system. 

The  pilots'  reactions  to  the  candidate  advanced  alerting  concepts  were  very 
positive.  They  stated  an  overall  preference  for  the  advanced  systems  in 
comparison  to  the  conventional  system.  They  said  that  the  advanced  systems 
were  easier  to  use  and  were  more  effective  than  the  conventional.  The  master 
aural  sounds  were  rated  very  high  with  respect  to  both  attention-getting 
quality  and  total  number  (3).  The  pilots  singled  out  the  centralized  location 
for  visual  Information  display,  the  unique  colors  and  sounds  for  each  urgency 
level,  the  use  of  voice,  and  the  volume  of  the  aural  components  as  features 
which  they  especially  liked.  The  only  feature  that  was  objected  to  by  several 
pilots  was  that  the  master  aural  caution  alert  used  sounded  too  urgent.  In 
the  final  analysis,  both  candidate  systems  concepts  were  validated,  and  a 
combination  of  the  features  of  the  two  systems  was  used  In  formulating  the 
design  guidelines. 

2.3.2  FLIGHT  ENGINEER  STATION  TESTS 

The  results  of  these  tests  validated  the  candidate  system  design  at  the  Flight 
Engineer  Station.  Shorter  response  times  and  fewer  missed  alerts  were 
obtained  with  the  advanced  system  In  comparison  to  the  conventional  system. 
Pilot  opinion  data  also  Indicated  a  strong  preference  for  the  advanced  system. 

2 A  TIME-CRITICAL  WARNING  TESTS 

Each  pilot  flew  twelve  flights  to  evaluate  time-critical  alerts.  Those  trials 
were  conducted  to  evaluate  alternative  presentation  media  and  display  formats. 
The  time-critical  tests  were  Identical  to  the  system  validation  tests  except 
only  warning  alerts  (time-critical  and  non- time-critical)  were  presented. 

Eight  alerts  were  presented  In  each  test  flight.  The  variables  Investigated 


•  Display  Location  -  In  the  pilots'  primary  or  secondary  field  of  view 


•  Presentation  Format  -  alphameric,  graphic,  or  both 

•  Massage  Content  -  status  or  guidance 

•  Accompanying  Voice  Alert  -  yes  or  no 

The  results  of  these  tests  are  summarized  below,  and  described  In  detail  In 
Section  5. 

Since  the  same  attention-getting  devices  were  used  for  both  the 
non- time-critical  and  the  time-critical  warnings,  no  measurable  difference  was 
found  In  the  mean  detection  times.  The  time-critical  mean  detection  times, 
however,  reveal  that,  when  status  Information  was  presented  graphically, 
significantly  longer  mean  detection  times  resulted  than  when  guidance 
Information  was  presented.  This  finding  suggests  that  the  status  messages  may 
cause  an  Increase  In  the  pilots  mental  workload. 

Performance  data  Indicated  that  the  pilots  used  alert  urgency  to  Influence 
their  response.  The  content  as  well  as  the  location  of  the  time-critical 
Information  proved  to  be  very  Important  to  their  response.  The  alerts  that 
provided  guidance  Information  graphically  In  the  pilot's  primary  field  of  view 
with  voice  resulted  In  the  shortest  mean  response  times.  On  the  other  hand, 
the  longest  mean  response  times  occurred  when  status  Information  was  presented 
graphically  In  the  pilot's  secondary  field  of  view,  even  though  voice  was 
present. 

The  pilots  preferred  the  guidance  Information  to  the  status  Information  for 
time-critical  alerts.  However,  they  felt  that  the  graphics  used  In  the  study 
were  too  cluttered.  They  believed  that  the  portion  of  each  graphic  which 
provides  the  guidance  Information  should  be  emphasized.  They  had  no 
preference  for  either  of  the  locations  used  In  the  test  but  said  that 
consideration  should  be  given  to  using  the  Electronic  Attitude  Direction 
Indicator  (EADI)  to  present  the  guidance  alerts. 
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2.S  FUTURE  DIRECTIONS 


This  stu<(y  has  dealt  with  those  system  functions  and  components  which  are 
directly  Involved  in  alerting  and  Informing  the  crew  of  abnormal  conditions  or 
situations.  These  functions  and  components  should  be  standardized  among  the 
different  aircraft  types.  However,  there  are  other  parts  of  the  total 
alerting  system  which  are  airplane- spec If 1c  and  were  not  Investigated  or 
resolved  In  this  study.  Included  In  this  larger  system  are  such  things  as: 
fault  sensors  and  sensor  combinations,  the  definition  and  categorization  of 
specific  alerts;  the  logic  used  to  prioritize  and  Inhibit  alerts;  the  logic 
for  multiple  failure  conditions,  and  the  allocation  of  computing  functions. 

Aircraft  or  operational -specific  considerations  should  be  Included  In  the 
alerting  system  design.  Examination  of  existing  accident/ Incident  data  could 
Identify  factors  contributing  to  accidents  and  Incidents,  and  the  role.  If 
any,  that  the  alerting  system  plays  should  be  assessed.  A  second  area  of 
consideration  Is  the  environment  In  which  the  system  must  function.  This 
Includes  such  operational  considerations  as:  alert  frequency;  crew  responses; 
alert  system  operation;  flight  deck  operations  and  activity;  crewnember 
responsibilities,  and  user  opinion  of  current  line-pilots  system  needs. 
Investigations  In  these  two  areas  could  assess  the  potential  for  the  alerting 
system  to  monitor  the  aircraft  for  malfunctions  and  failures,  to  anticipate 
problems,  and  expedite  resolution  of  problems. 

Another  area  which  requires  further  Investigation  Is  the  formatting  of  the 
time-critical  alerts.  Improperly  designed  displays  and  display  formats  can 
confuse  and  Impede  pilot  responses,  whereas  proper  designs  can  have  the 
opposite  effect.  Near-term  requirements  for  time-critical  display  of 
collision  avoidance,  wlndshear  and  perhaps  active  control  failures  are 
anticipated. 

Alert  prioritization  and  Inhibition  schemes  can  promote  more  efficient 
Information  handling.  Both  are  highly  dependent  on  the  specific  aircraft 
type.  Methods  and  techniques  for  developing  these  schemes  should  be 
Identified  to  aid  In  future  system  design. 


Computer  and  display  technology  have  reached  a  point  where  the  manipulation 
and  presentation  of  checklist  and  procedural  Information  should  be  reviewed 
with  the  prospect  of  Improving  crew  response  to  emergencies.  Areas  of 
Interest  Include  display  format,  checklist  accessabillty,  and  the  possibility 
of  providing  checklist  Information  formulated  directly  from  computer  logic  to 
aid  the  crew  In  responding  to  multiple  failure  situations. 

Finally,  flight  test  validation  should  be  used  to  further  refine  these  guide¬ 
lines. 


3.0  TEST  FACILITY 


The  various  study  requirements  dictated  the  use  of  a  facility  In  which  a 
flight  deck  system  could  be  Integrated,  tested  and  evaluated  In  a  simulated 
environment.  This  facility  consists  essentially  of  a  generic  cab  that  serves 
as  an  "operational  breadboard"  to  facilitate  the  development  of  flight  deck 
system  concepts,  functional  capabilities,  and  Interface  features.  Proposed 
systems,  system  changes,  and  alternative  mechanizations  can  be  evaluated  and 
demonstrated  In  such  a  facility.  It  also  provides  a  flexible  experimental 
simulation  laboratory  that  allows  for  easy  Introduction  of  new  hardware  and 
change  to  the  flight  deck  system  configuration.  System  software  Is 
modularized  to  facilitate  change;  Interface  equipment  Is  flexible  and  thus 
allows  for  wide  varieties  of  engineering  developmental  evaluations.  These 
elements  have  been  designed  Into  the  Boeing  Company's  Kent  Flight  Simulation 
Center  and  the  Douglas  Aircraft  Digital  Equipment  Technology  Analysis  Center 
(DETAC).  See  Figures  3.0-1  and  3.0-2  for  Illustrations  of  these  facilities. 
For  more  detailed  descriptions  refer  to  Appendix  A. 


Figure  3.0-2.  Douglas  DETA  C  Hardware  Configuration  for  Aircraft  Alerting 
Systems  Standardization  Study 


4.0  SUPPLEMENTAL  SIMULATOR  TESTS 

4.1  INTRODUCTION 

A  number  of  unresolved  Issues  were  Identified  during  Phase  1  of  this  study 
that  needed  to  be  resolved  prior  to  Phase  3  testing.  In  view  of  this,  a  set 
of  supplemental  tests  were  conducted  and  the  results  were  incorporated  into 
Phase  3  testing. 

Each  of  the  supplemental  tests  belonged  in  one  of  two  categories.  The  first 
involved  unresolved  issues  related  to  system  displays  and  controls;  the  second 
addressed  unresolved  issues  relating  to  system  logic.  Emphasis  was  given  to 
alert  prioritization,  flight  phase  inhibit  logic,  alert  formatting  and 
Interactive  functions. 

Analytical  data  was  collected  from  pilots  experienced  in  both  commercial 
airline  procedures  and  engineering/flight  test  operations. 

4.1.1  SYSTEM  COMPONENT  TESTS 

The  tests  conducted  in  Phase  1  identified  several  cases  where  more  than  one 
viable  option  in  system  component  design  selection  did  exist.  These 
alternative  designs  were  presented  to  a  number  of  experienced  pilots  in  a 
simulator  environment.  Subsequent  to  that  exposure,  pilots  were  asked  to 
evaluate  each  alternative,  relative  to  several  performance  criteria. 

4.1.1. 1  MASTER  VISUAL  ALERT-WARNING  AND  CAUTION 

The  first  test  addressed  the  master  visual  alert.  Some  current  commercial 
aircraft  employ  separate  master  warning  and  master  caution  lights,  while 
others  use  a  single  split  master  light  to  annunciate  both  warning  and  caution 
level  alerts  {the  top  half  of  the  switch  for  warning;  the  bottom  for  caution). 
The  alerting  system  philosophy  under  consideration  called  for  the  master  light 
to  act  not  only  as  an  attention-getting  device,  but  also  as  a  means  by  which 
the  alert  could  be  acknowledged  and/or  cancelled.  The  Issue  was  that  the 
split  master  light  is  desirable  from  a  space-efficient  point  of  view;  however. 
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the  possibility  that  multiple  alerts  could  be  cancelled  Inadvertently 
represents  a  potential  problem  Inherent  in  this  configuration. 

4.1. 1.2  MASTER  ALERT-ADVISORY 

The  second  test  Involved  the  presence/absence  of  a  visual  master  advisory 
alert.  By  definition,  advisory  level  information  requires  crew  awareness  but 
no  time  frame  Is  defined  for  response;  consequently  a  master  visual  alert  may 
not  be  required.  Pilots  were  asked  if  they  felt  that  advisory  level 
information  warrants  annunciation  In  their  primary  field  of  view.  If  they 
considered  the  presence  of  a  master  visual  advisory  to  be  an  unnecessary 
distraction,  it  would  not  be  incorporated  into  the  Phase  3  tests.  However,  if 
the  majority  of  pilots  stated  that  advisory  level  annunciation  is  needed,  a 
master  advisory  light  would  be  used  for  Phase  3  testing. 

The  issue  of  master  aural  alert  for  advisory  information  was  approached  in 
much  the  same  manner  as  that  for  the  master  advisory  light.  Pilot 
recommendations  for  appropriate  combinations  of  master  visual  and  aural  alerts 
were  also  solicited. 

4.1. 1.3  PRESENTATION  OF  ADVISORY  INFORMATION 

The  third  issue  addressed  was  the  methodology  to  be  used  for  the  display  of 
advisory  level  information.  Three  options  were  presented:  A  combination  of 
voice  and  alphanuneric  display  or  either  one  separately.  If  an  alphanumeric 
display  were  used,  advisory  level  alerts  would  be  presented  on  the  same 
Cathode  Ray  Tube  (CRT)  as  would  warning  and  caution  level  Information. 

4. 1.1. 4  INTERACTIVE  FUNCTIONS 

The  fourth  area  Investigated  concerned  selected  alerting  system  Interactive 
functions.  One  of  these  Involves  the  methodology  used  to  store  and  recall 
messages  from  memory.  Total  store/recall  may  be  a  desirable  feature  as  it 
would  reduce  the  Interactive  requirements  of  the  system.  However,  situations 
may  arise  where  a  selective  store/ recall  capability  Is  required.  Pilots  were 
asked  if  one  of  the  two  alternatives  could  handle  all  flight  crew  Information 
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requirements  or  if  both  were  necessary.  The  second  interactive  function 
investigated  was  the  methodology  by  which  procedural  information  should  be 
obtained.  Pilots  were  asked  in  what  situations  procedural  information  should 
be  provided,  the  level  of  detail  required  and  the  most  appropriate  mode  of 
presentation.  The  third  interactive  function  involved  the  attenuation/ 
cancellation  of  aural  alerts  (tones  and  verbal  messages).  It  was  first 
necessary  to  determine  whether  aural  alerts  should  be  cancelled  or  merely 
attenuated.  After  this  issue  was  resolved,  it  was  then  necessary  to  determine 
whether  the  attenuation  and/or  cancellation  should  be  done  manually  or 
automatically. 

4.1. 1.5  VERBAL  MASTER  ALERT 

The  last  test  centered  on  the  use  of  master  verbal  versus  sound  alerts.  It 
has  been  suggested  that  the  words  "warning"  and  "caution"  may  be  more 
effective  as  precursors  to  corresponding  alert  messages  than  would  discrete 
warning  and  caution  sounds.  Again,  pilots  were  asked  to  cite  specific 
operational  advantages  and  disadvantages  that  might  be  associated  with  each  of 
these  alternatives. 

4.1.2  SYSTEM  LOGIC  TESTS 

In  addition  to  the  issues  regarding  system  components,  a  number  of  questions 
were  raised  relative  to  the  logic  by  which  these  components  should  operate. 
These  questions  are  presented  here  along  with  the  general  methodology  that  was 
used  to  obtain  the  answers. 

4.1.2.1  PRIORITIZATION 

It  has  been  suggested  that  the  gross  categorization  of  alert  messages  into 
three  main  categories  (warning,  caution  and  advisory)  is  not  sufficient  for 
conveying  essential  information  to  the  flight  crew.  It  is  felt  by  some  that 
prioritization  of  alerts  within  the  three  categories  would  provide  valuable 
Information  which  could  be  used  as  an  aid  in  selecting  the  most  critical 
problem  to  be  addressed.  The  prioritization  would  be  used  primarily  on  the 
visual  information  display.  Described  simply,  the  messages  in  each  alert 
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level  would  be  automatically  prioritized  by  the  system  computer  and  when  a 
particular  fault  occurred,  the  message  would  appear  on  the  display  in  its 
appropriate  position  relative  to  the  other  messages  of  the  same  category 
already  present.  With  this  arrangement,  the  most  important  message  would 
always  be  at  the  top  of  the  display,  regardless  of  time  of  occurrence.  This 
is  the  key  advantage  for  prioritization  within  gross  category  level.  Flight 
operations  personnel  from  Douglas,  Lockheed  and  Boeing  were  employed  to  assist 
in  the  development  of  a  questionnaire  to  be  used  as  an  aid  in  prioritizing 
al erts . 

4.1. 2.2  INHIBIT  LOGIC 

Another  issue  addressed  was  the  subject  of  flight  phase  inhibit  logic.  This 
capability  would  allow  the  system  to  inhibit  non-essential  information  during 
high  workload  flight  segments.  Presently,  there  is  little  agreement  as  to 
which  alerts  should  be  inhibited  during  each  flight  phase. 

4. 1.2.3  VISUAL  MESSAGE  SYNTAX 

A  third  area  of  interest  was  visual  message  syntax.  To  date,  no  systematic 
effort  has  been  made  to  verify  the  consistency  of  alert  message  formatting. 

The  objective  was  to  survey  the  industry  to  identify  the  most  prevelant  alert 
formatting  methodology,  if  one  exists.  It  was  anticipated  that  the  vast 
majority  of  alert  messages  would  be  structured  with  the  nature  of  the  problem 
preceding  the  location  or  vice  versa.  Information  obtained  from  this  survey 
was  summarized  and  presented  to  the  pilots.  They  were  then  asked  to  recommend 
a  specific  format  and  to  explain  their  selection  rationale. 

4.1. 1.4  MULTIPLE  VERBAL  ALERTS 

The  fourth  issue  addressed  involved  the  sequencing  of  multiple  verbal  alerts. 

A  number  of  alternatives  were  presented.  One  of  these  prioritizes  the  alert 
messages  and  annunciates  only  the  most  severe  problem.  Subsequent  message! s) 
would  be  annunciated  only  after  more  serious  one(s)  had  been  corrected  or 
somehow  accommodated.  A  second  alternative  would  be  to  introduce  the  most 
important  message  for  a  fixed  number  of  repetitions,  cancel  it  and  introduce 
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each  subsequent  message  in  a  similar  manner.  A  third  alternative  was  to 
annunciate  the  message  "multiple  alerts"  and  direct  crew  attention  to  the 
visual  Information  display  for  the  specific  fault  messages.  Again,  the 
Intention  was  to  have  experienced  pilots  evaluate  each  of  the  alternatives 
and,  if  possible,  generate  additional  alternatives  for  subsequent  evaluation 

4.2  METHODOLOGY 

4.2.1  PILOT  SAMPLE 

A  total  of  25  pilots  participated  in  the  simulator  evaluation.  They  were 
selected  from  5  organizations  representing  both  airframe  manufacturers  and 
customer  airlines.  Collectively,  these  pilots  had  experience  on  aircraft 
built  by  each  of  the  major  U.S.  airframe  manufacturers.  Table  4. 2. 1-1  shows 
the  organization  and  aircraft  type  represented  as  well  as  the  mean  number  of 
flight  hours  for  the  25  pilots. 


Table  4.2. 1-1.  Organizations,  Aircraft  Types,  and  Flight-Hours  of  Pilots 
Participating  in  Supplemental  Simulator  Evaluation 


(a)  Organization*  Represented 


Organization 

Number  of  pilots 

Douglas  Aircraft  Company 

S 

Boeing  Commercial  Airplane  Company 

2 

Lockheed  California  Company 

4 

Continental  Airlines 

6 

Western  Airlines 

8 

Total 

25 

(b)  Aircraft  Type*  Represented  (Vehicle  of  Mott  Racant  Experience) 


Aircraft  types 

Number  of  pilots 

DC-9 

2 

DC-10 

6 

8727 

12 

B747 

1 

L1011 

4 

Note:  Mean  number  of  flight-hours:  1 1 ,319 
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4.2.2  FACILITY 


The  supplemental  tests  were  conducted  In  the  Digital  Equipment  Technology 
Analysis  Center  (DETAC).  Several  changes  had  to  be  Incorporated  Into  the 
DETAC  facility  to  accommodate  this  activity. 

A  split  legend  master  warning  and  caution  light  was  added  to  the  glareshleld 
on  the  First  Officer's  side  and  a  master  advisory  light  was  Installed  on  the 
Captain's  side.  Software  modifications  were  made  so  that  both  of  these 
switches  would  be  operational.  Additional  software  changes  were  made  to 
provide  a  number  of  Interactive  functions  which  Included: 

•  selective  and  total  store/recall 

•  access  to  procedural  Information 

•  line  advance  capability 

•  attenuation/cancellation  of  verbal  alerts 

A  more  complete  description  can  be  found  In  Appendix  A. 

4.2.3  PROCEDURE 

Two  pilots  participated  in  each  session  which  lasted  approximately  2  hours. 
Most  of  this  time  was  spent  in  the  simulator  while  the  remainder  was  used  in  a 
debriefing  session  In  an  area  outside  the  DETAC  facility. 

Prior  to  entering  the  cockpit,  the  pilots  were  given  a  brief  summary  of  the 
work  done  to  date  on  the  project.  Since  a  majority  of  the  participants  were 
not  closely  associated  with  research,  this  type  of  description  provided  them 
with  a  perspective  from  which  to  evaluate  the  concepts  In  question. 

Upon  entering  the  DETAC  facility,  the  pilots  were  asked  to  seat  themselves  In 
the  Captain's  and  First  Officer's  seats  so  that  both  could  easily  see  the 
systems  to  be  evaluated.  The  briefing  began  with  an  overview  of  development 
of  crew  alerting  systems.  Then  a  number  of  disadvantages  Inherent  in  current 
alerting  system  design  were  described.  Particular  emphasis  was  given  to  the 
annunciator  matrix  which  Is  present  on  most  current  aircraft  flight  decks. 


This  overview  was  followed  by  a  discussion  of  the  potential  advantages 
associated  with  an  advanced  alerting  system,  several  components  of  which  were 
demonstrated  during  the  simulator  sessions. 

The  following  features  of  the  alerting  system  were  demonstrated  to  the  pilots: 

•  New  Message  Indicator:  Identifies  and  annunciates  the  most  recent 
message  on  the  display. 

•  Line  Advance  Function:  Provides  a  line  address  capability  which  enables 
an  operator  to  call  up  procedural  Information  for  a  particular  alert.  It 
also  provides  the  capability  to  store  messages  selectively  In  memory. 

•  Memory  and  Store/Recall  Capability:  Non-crltlcal  fault  messages  could  be 
placed  In  memory  by  one  of  2  methods:  all  messages  stored  simultaneously 
or  selectively. 

•  Deferred  Item  Indicator:  Serves  to  Inform  the  flight  crew  that  fault 
messages  are  stored  In  memory. 

•  Page  Indicator  and  Paging  Function:  Provides  an  Indication  that  a 
display  overflow  has  occurred.  When  the  number  of  alerts  exceeds  the 
display  capability,  the  software  system  automatically  provides  additional 
"pages"  for  the  excess  faults.  The  page  Indicator  denotes  which  page  Is 
being  viewed. 

•  Overflow  Information  Indicators:  Informs  the  operator  of  the  status  of 
an  overflow  condition,  l.e.,  how  many  message  lines  are  on  the  next  page. 

•  Procedure/Checklist  Function:  Provides  the  capability  to  display 
Information  to  be  used  to  correct  the  selected  failure. 

The  pilots  were  given  the  opportunity  to  operate  each  of  these  functions  to 
familiarize  themselves  with  the  control /display  dynamics.  Illustrations  of 
these  features  are  contained  In  Appendix  B. 


Two  fault  monitoring  capabilities  were  also  demonstrated.  The  first  entailed 
placing  a  dispatch  Inoperative  Item  Into  memory  and  Illustrating  how  the 
system  would  constantly  monitor  It  during  flight.  As  aircraft  status  changes 
the  alert  placed  in  memory  would  automatically  be  brought  back  to  page  one  of 
the  display.  The  second  feature  demonstrated  was  the  capability  of  the  system 
to  handle  flight  phase  prioritization  and  was  illustrated  by  showing  the 
pilots  how  a  specific  fault  (hydraulic  failure)  was  automatically 
re-prioritized  as  the  aircraft  transitioned  from  descent  to  final  approach. 

The  next  function  demonstrated  was  the  means  by  which  voice  alerts  could  be 
attenuated  or  cancelled.  Voice  alerts  were  presented  and  the  pilots  were 
allowed  to  manually  cancel  and  attenuate  the  alerts  on  two  successive  trials. 
They  also  observed  while  the  system  automatically  cancelled  and  attenuated  the 
alerts  during  two  additional  trials. 

The  participants  were  then  given  a  demonstration  of  the  system  control  options 
for  the  master  visual  alert.  This  was  preceded  by  a  brief  suninary  of  the 
split  and  separate  master  visual  alerts.  Also  demonstrated  were  the  location, 
function  and  operation  of  the  master  visual  advisory  alert. 

Finally,  various  display  formats  currently  under  consideration  (see  Appendix 
E)  were  shown  to  the  pilots  on  the  display  screen.  Each  format  was 
accomplished  by  a  graphic  representation  of  the  four  formats  for  purposes  of 
comparl son . 

Pilots  were  encouraged  to  ask  questions  or  make  comments  at  any  time.  They 
were  also  offered  repeated  demonstrations  of  any  areas  that  they  felt  required 
additional  review.  All  relevant  pilot  comments  were  recorded  for  subsequent 
documentation. 

Following  the  simulator  session,  participants  were  Invited  to  a  debriefing 
area  where  they  were  given  a  questionnaire  (debriefing  summary)  to  complete. 
The  questionnaire  was  structured  to  solicit  pilot  opinion  for  each  of  the 
Issues.  A  number  of  Issues  were  addressed  In  the  questionnaire  that  were  not 
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demonstrated  In  the  simulator.  The  first  of  these  was  the  Importance  of  a 
master  advisory  tone.  They  were  also  asked  about  visual  Information  display 
message  syntax  and  a  verbal  master  aural  alert.  The  advisory  tone  and  verbal 
alert  were  not  demonstrated  In  the  simulator  because  It  was  felt  that  the 
pilots  would  tend  to  evaluate  the  actual  sounds  (tone  and  words)  rather  than 
the  concepts  behind  them.  Illustration  of  alternative  visual  message  syntaxes 
was  restricted  to  the  debriefing  summary  because  demonstration  In  the  DETAC 
facility  would  not  have  provided  the  pilots  with  any  Information  or 
perspective  over  and  above  that  provided  by  the  questionnaire. 

Upon  completion  of  the  debriefing  sumnary,  an  informal  discussion  took  place 
wherein  the  pilots  asked  questions  and  made  comments  about  various  alerting 
system  and  aircraft  Issues.  Again,  comments  relating  to  the  development  of 
advanced  alerting  systems  were  recorded. 

Before  leaving,  the  pilots  were  given  two  additional  questionnaires  which  they 
were  asked  to  complete  and  return  at  their  earliest  convenience.  One  of  these 
was  a  supplemental  debriefing  questionnaire  designed  to  gather  additional 
information  on  the  alerting  system  display  characteristics  currently  under 
consideration.  The  other  questionnaire  was  concerned  with  alert 
prioritization  and  Inhibition.  Results  from  this  questionnaire  were  used  to 
help  determine  the  feasibility  of  a  prioritization  scheme  and  to  gain  some 
Insight  Into  the  problems  that  might  be  encountered  during  development  and 
Implementation  efforts. 

4.3  RESULTS 

4.3.1  DEBRIEFING  SUMMARY 

A  copy  of  the  debriefing  summary  and  the  pilot  responses  are  contained  In 
Appendix  C. 

Each  of  the  25  pilots  completed  the  debriefing  summary.  Table  4. 3. 1-1 
presents  a  summary  of  pilot  preferences  for  each  of  the  Issues  addressed  along 
with  the  results  of  the  chi  square  analysis.  Each  Issue  Is  also  addressed 
separately  In  the  following  sections. 


Table  4.3. 1 *  1.  Summary  of  Issues,  Options,  and  Significant  Pilot  Preferences 


Issue 

Options 

Pilot 

preferences 

D 

Significance 

level 

Comments 

Master  alert* 

Split  matter 

11 

0.36 

Not 

(warning  and  caution) 

Separate  mester 

14 

significant 

Master  alert 

Master  lijht 

4 

25.11 

P<  0.001 

(advisory) 

Discrete  tone 

0 

Voice 

0 

Flashing  box 

3 

Master  light  and  tone 

1 

Master  light  and  voice 
Master  light  and 

2 

flashing  box 

12 

Tone  and  flashing  box 

2 

Other 

1 

Advisory 

Voice 

0 

17.64 

P<0j001 

information 

Alphanumeric  display 

23 

display 

Voice  plus  display 

2 

Other 

0 

Store/recall 

Total  store/recall 

8 

7.21 

P<  0.05 

Difference  between 

capability 

Selective  store/recall 

3 

"total"  and  "combi- 

Combination  (total  and 

nation"  not 

selective) 

14 

significant 

Other 

0 

Visual  message 

Gen  heading-subsystem- 

syntax 

nature  of  emer 

Nature  of  emer- 

20 

32.76 

P<  0.001 

general  heading- 
subsystem 

3 

Other 

2 

Voice  alert 

Words  better  than  tones 

15 

11.84 

P<0.01 

Difference  between 

Tones  better  than  words 

9 

"tones  better"  and 

No  difference 

1 

"words  better"  not 
significant 

Multiple  verbal 

Prioritize  messages 

18.03 

P<  0.001 

Difference  between 

alerts** 

Cancel  after  correction 

8 

two  most  preferred 

Prioritize  messages 

options  not 

Cancel  after  fixed 

0 

significant 

number  of  repetitions 
Multiple  alerts 

14 

Other 

3 

Cancellation/ 

Automatic  attenuation 

2 

36.0 

P<  0.001 

attenuation 

Manual  attenuation 

0 

of  aural  alerts** 

Automatic  cancellation 

4 

Manual  cancellation 

19 

*  The  pilots  did  not  show  a  clear  preference  for  either  the  split  or  separate  master  lights. 

Fourteen  preferred  seperate  matter  lights;  the  remainder  (It)  favored  the  split  concept 

**  No.  1  rankings. 


43.1.1  MASTER  VISUAL  ALERT-WARNING  AND  CAUTIONS 

For  warning  and  caution  level  alerts,  the  pilots  did  not  show  a  clear 
preference  for  either  the  split  or  separate  master  light.  While  a  majority 
preferred  the  separate  master  visual  Indicator  switch,  this  difference  was  not 
statistically  significant. 

4.3. 1.2  MASTER  ALERT-ADVISORY 

There  was  a  decided  preference  (12)  for  a  master  advisory  light  In  conjunction 

with  a  flashing  box  around  the  most  recent  message.  A  chi  square  test  for 
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goodness  of  fit  showed  that  this  preference  was  highly  significant  (X  =  25.11 
<  .001).  This  was  the  only  option  preferred  by  more  than  4  pilots. 

4.3.1.3  PRESENTATION  OF  ADVISORY  INFORMATION 

A  vast  majority  of  the  pilots  (23  of  25)  preferred  the  al phanunerlc  display 
over  voice  or  a  combination  of  the  two  to  annunciate  advisory  Information. 
Again,  this  preference  was  highly  significant  (X2  *  17.64;p  <  .001). 

4.3.1. 4  INTERACTIVE  FUNCTIONS 

When  questioned  about  system  memory  requirements,  14  pilots  preferred  a 

combination  of  total  and  selective  store/recall.  A  chi  square  test  for 

goodness  of  fit  between  the  three  options  listed  showed  this  preference  to  be 

o 

statistically  significant  (X  =  7.21;  p  <  .05).  When  a  chi  square  was 
computed  for  the  two  most  frequent  choices,  the  result  was  not  statistically 
significant. 

4.3. 1.5  PROCEDURAL/CHECKLIST  INFORMATION 

When  asked  to  judge  the  automatic  checklist/procedural  Information  features,  a 
significant  majority  (23  of  25)  felt  that  this  capability  would  be  good  to 
excellent.  This  question  elicited  a  number  of  useful  suggestions  from  the 
pilots  that  could  be  Implemented  In  future  work  on  procedural/checklist 
Information  displays,  (See  Appendix  B). 


*•3.1.6  ATTENUATION/CANCELLATION  OF  AURAL  ALERTS 

The  clear  preference  (19  out  of  25)  was  for  manual  cancellation.  This 
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preference  was  highly  significant  (X  =  36.0;  p  <  .001).  The  remaining  6 
pilots  preferred  either  automatic  volume  reduction  (2)  or  automatic 
cancellation  (4)  after  a  fixed  number  of  repetitions. 

4.3.1. 7  VISUAL  MESSAGE  SYNTAX 

Regarding  visual  message  syntax,  a  majority  of  the  respondents  (20  out  of  25) 
preferred  the  format  that  showed  the  general  heading  followed  by  the  subsystem 
and  nature  of  the  emergency.  This  highly  significant  preference  (X2  =  32.76; 
p  <  .001)  may  be  explained  in  part  by  the  fact  that  the  most  preferred  option 
was  also  the  example  provided  in  the  question  (see  Appendix  B). 

4.3. 1.8  VERBAL  MASTER  ALERT 

A  majority  of  the  pilots  (15  out  of  28)  favored  words  rather  than  a  discrete 
sound  for  the  aural  master  alert.  This  preference,  however,  was  not 
statistically  significant. 

4.3.1.9  MULTIPLE  VERBAL  ALERTS 

Most  pilots  (14  out  of  25)  preferred  introduction  of  the  message  "multiple 
alerts".  Although  the  overall  difference  in  preference  between  the  four 
options  was  highly  significant  (X2  =  18.03;  p  <  .001),  the  difference  between 
the  two  most  preferred  options  was  not  significant  (where  the  second  most 
preferred  option  was  to  prioritize  messages  and  annunciate  only  the  most 
severe  problem  and  annunciate  subsequent  message(s)  only  after  the  previous 
one  had  been  corrected  or  somehow  accommodated). 

4.3.2  SUPPLEMENTAL  QUESTIONNAIRE 

All  pilots  who  participated  In  the  supplemental  simulator  evaluations  were 
asked  to  rate  a  number  of  additional  system  capabilities  and  characteristics 
which  are  shown  In  Appendix  B. 
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A  total  of  17  pilots  completed  and  returned  the  supplemental  questionnaire. 

The  questionnaire  along  with  the  mean  ratings  for  each  question  can  be  found 
In  Appendix  0.  Table  4. 3. 2-1  shows  the  organizations  represented  In  this 
group,  along  with  the  mean  number  of  flight  hours  and  the  most  recent  aircraft 
experience  of  the  pilots  surveyed. 

Four  display  formats  were  evaluated,  which  can  be  described  as  follows: 

1.  Reverse  chronology  within  color:  New  message  appears  at  the  top  of  Its 
alert  group  (color  coded  urgency  level). 

2.  Priority:  New  message  appears  at  a  pre-determined  location  within  Its 
alert  group  depending  on  Its  priority  level. 

3.  Chronology  within  color:  New  message  appears  at  the  bottom  of  Its  alert 
group . 

4.  Chronology:  Most  recent  message  appears  at  the  top  of  the  list, 
regardless  of  color  or  specific  priority  level. 

Figure  4. 3. 2-1  Illustrates  how  the  display  would  look  using  each  of  the 
alternative  formats.  Two  failures  from  each  urgency  level  (warning,  caution, 
advisory)  occurred  In  the  following  order: 

1.  Gen  Bus  3  Fall  -  Caution  (Amber)  -  (First  failure  to  occur) 

2.  AC  Bus  3  Off  -  Caution  (Amber) 

3.  Pack  3  Off  -  Advisory  (Blue) 

4.  Cabin  Altitude  -  Warning  (Red) 

5.  L.  Emer.  AC  Bus  -  Warning  (Red) 

6.  Flap  Limit  Inop  -  Advisory  (Blue)  -  (Last  failure  to  occur) 

The  pilots  were  asked  to  evaluate  4  display  formats  (Figure  4. 3.2-1)  relative 
to  5  separate  questions.  As  can  be  seen  In  Figure  4. 3. 2-2,  they  considered  4 
of  the  5  questions  to  be  somewhat  to  very  Important.  Relative  to  these  4 
criteria,  the  trend  shows  a  preference  for  the  format  employing  reverse 
chronology  within  color.  This  was  not,  however,  a  significant  preference,  as 
Illustrated  by  the  overlap  of  the  951  confidence  limits. 
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Table  4.3.2-1.  Organizations,  Aircraft  Types,  and  Flight-Hours  of  Pilots 
Completing  the  Supplemental  Questionnaire 


(a)  Organizations  Rapreaantad 


Organization 

Number  of  pilots 

Douglas  Aircraft  Company 

4 

Boeing  Commercial  Airplane  Company 

2 

Lockheed  California  Company 

4 

Continental  Airlines 

4 

Western  Airlines 

3 

Total 

17  j 

(b)  Aircraft  Typas  Rapraaantad  (Vehicle  of  Most  Recent  Experience) 

Aircraft  type 

Number  of  pilots 

DC-9 

4 

DC-10 

1 

B727 

8 

L-1011 

4 

Note'.  Mean  number  of  flight-hours:  13,019 


Reverse  chronology  within  color 

Chronology  within  color 

(R)  L  EMER  AC  BUS 
(R)  CABIN  ALTITUDE 
(A)  AC  BUS  3  OFF 

(A)  GEN  BUS  3  FAIL 

(B)  FLAP  LIMIT  INOP 
(B)  PACK  3  OFF 

(R)  CABIN  ALTITUDE 
(R)  L  EMER  AC  BUS 
(A)  GEN  BUS  3. FAIL 

(A)  AC  BUS  3  OFF 

(B)  PACK  3  OFF 

(B)  FLAP  LIMIT  INOP 

Priority 

Chronology 

(R)  L  EMER  AC  BUS 
(R)  CABIN  ALTITUDE 
(A)  AC  8US  3  OFF 

(A)  GEN  BUS  3  FAIL 

(B)  PACK  3  OFF 

(B)  FLAP  LIMIT  INOP 

(B)  FLAP  LIMIT  INOP 
(R)  L  EMER  AC  BUS 
(R>  CABIN  ALTITUDE 
(B)  PACK  3  OFF 
(A)  AC  BUS  3  OFF 
(A)  GEN  BUS  3  FAIL 

Legend: 


(R)  Red 
(A)  Amber 
<B)  Blue 


Figure  4.3.2- 1.  Display  Formats 


Key: 


MIAN  IMPORTANCE 
RATING: 


1:  EXCELLENT- 
NO  CHANGES 
RECOMMENDED 


2:  GOOD- 
MINOR  CHANGES 
BENEFICIAL 


3:  FAIR- 
MINOR  CHANGES 
RECOMMENDED 


4:  POOR- 
MAJOR  CHANGES 
RECOMMENDED 


6:  VERY  POOR- 
MAJOR  CHANGES 
NECESSARY 


Import! no  rating  tola 
1 :  Very  important 
2:  Somewhat  important 

3:  Neither  important  nor 
unimportant 


4:  Somewhat  unimportant 
S:  Very  unimportant 


Display  format 

□  Chronology  within  color 

•  Reverse  chronology  within 
color 

O  Priority 

*  Basic  chronology 


Figure  4.3.2-2.  Mean  Pilot  Ratings  for  Overflow  Logic  Options  (With  95%  Confidence  Limits ) 
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Pilot  judgements  were  solicited  on  a  number  of  other  display  characteristics. 

The  results  can  be  summarized  as  follows: 

•  Overall  character  word  and  line  spacing  was  judged  as  being  good  to 
excellent. 

•  Stroke  width  was  fair  to  good. 

•  The  contrast  between  the  red  (warning)  and  amber  (caution)  messages,  and 
the  display  screen  was  good  to  excellent. 

•  The  blue  (advisory)  messages  and  the  white  ancilary  Information  (mode, 
page  and  deferred  Item  Indicators)  were  fair  to  good  relative  to 
background  brightness. 

•  The  location,  color  and  meaning  of  the  alert/memory  mode  indicator  was 
judged  as  being  fair  to  good. 

•  Pilots  preferred  the  colored  box  over  the  asterisk  as  a  new  message 
indicator. 

•  The  page  number  indicator  was  fair  to  good  in  terms  of  location,  size, 
clarity  of  meaning  and  contrast  with  alert  messages. 

•  The  cursor  (line  advance  indicator)  was  good  to  excellent  relative  to 
size  and  contrast  with  alert  messages  while  it  was  seen  as  being  fair  to 
good  in  terms  of  shape  and  key  input  dynamics. 

•  The  size,  shape,  location,  contrast,  and  flash  rate  of  the  colored  boxes 
used  to  indicate  the  presence  of  an  overflow  condition  were  judged  as 
being  good.  In  terms  of  clarity,  they  were  seen  as  being  fair  to  good. 

•  The  deferred  item  indicator  (in  memory  symbol)  size,  shape,  location  and 
clarity  of  meaning  was  viewed  as  being  fair  to  good. 
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•  Pilots  rated  the  function  keys  as  good  to  excellent  in  terms  of  size 
shape  and  location  while  contrast  and  correspondence  of  display  dynamics 
to  key  inputs  were  judged  as  being  fair  to  good. 

•  The  line  keys  were  rated  as  being  fair  to  good  relative  to  size,  shape, 
location,  contrast  and  display  dynamics. 

•  The  use  of  color  to  differentiate  between  warning,  caution  and  advisory 
messages  was  judged  as  being  good  to  excellent  in  terms  of  clarity  and  as 
an  aid  in  evaluating  aircraft  status. 

4.3.3  ALERT  PRIORITIZATION  AND  INHIBIT  QUESTIONNAIRE 

A  total  of  21  pilots  from  5  organizations  completed  and  returned  this 
questionnaire.  Table  4. 3. 3-1  shows  the  organizations  which  these  pilots 
represented  as  well  as  their  most  recent  aircraft  experience  and  the  mean 
number  of  flight  hours  for  the  group. 

4.3.3. 1  ALERT  PRIORITIZATION 

Figure  4.3.3. 1-1  illustrates  the  questionnaire  form.  All  information  required 
to  complete  the  questionnaire  is'  contained  on  the  form.  Mean  pilot  ratings 
for  2  of  the  16  alerts  ("Antiskid  Left  Inboard  Failure"  and  "APU  Fire")  are 
illustrated  in  Figures  4.3.3.4-2a  and  -2b  respectively.  In  Figure  4.3.3-2a 
(Antiskid  Left  Inboard  Failure"),  priority  ratings  for  flight  segments  D  and  E 
(initial  climb  and  climb-cruise-descent)  were  significantly  lower  than  for  the 
other  flight  phases;  the  95$  confidence  limits  for  flight  phases  D  and  E  do 
not  overlap  with  those  for  the  other  flight  segments.  The  point  to  be  made 
here  is  that  flight  phase  inhibit  logic  may  be  necessary  when  alert  priority 
is  not  consistent  across  flight  phases.  Figure  4.3.3-2b,  however,  shows  that 
the  pilots  rated  the  priority  of  the  "APU  Fire"  alert  consistently  across  all 
flight  phases.  Appendix  E  presents  the  information  obtained  from  this 
questionnaire.  Additional  examples  of  overlapping/ non-overlapping  priorities 
can  be  found;  also  to  be  found  are  alerts  that  were  rated  as  cautions  in  some 
flight  phases  and  warnings  in  others. 
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Table  4.3.3-1.  Organizations.  Aircraft  Types,  and  Flight-Hours  of  Pilots  Completing 
the  Alert  Prioritization  and  Inhibit  Questionnaire 


(a)  Organization  Represented 


Organization 

Number  of  pilots 

Douglas  Aircraft  Company 

7 

Boeing  Commercial  Airplane  Company 

2 

Lockheed  California  Company 

3 

Continental  Airlines 

4 

Western  Airlines 

_5_ 

Total 

21 

(b)  Aircraft  Types  Represented  (Vehicle  of  Mott  Recent  Experience) 


Aircraft  type 

Number  of  pilots 

DC-9 

3 

DC-10 

5 

B727 

9 

B747 

1 

L-1011 

3 

Note:  Mean  number  of  flight-hours:  1 1,669 


Across  the  8  flight  phases,  13  of  the  16  alerts  were  rated  as  cautions  by  the 
pilots.  Figure  4. 3. 3. 1-3  shows  how  the  pilots  rated  these  16  alerts  for  the 
final  takeoff  phase.  In  looking  at  the  95%  confidence  limits.  It  Is  apparent 
that  no  clear  overall  preference  was  given  for  prioritizing  the  alerts. 

Ratings  of  the  alerts  for  the  other  seven  flight  segments  can  be  found  In 
Appendix  E. 

4.3.3.2  INHIBIT  LOGIC 

Table  4. 3. 3. 2-1  summarizes  the  Inhibit  preferences  for  all  16  alerts  across 
the  8  flight  phases  as  well  the  percentage  of  pilots  who  saw  the  further  need 
to  consider  aircraft  configuration  In  the  Inhibit  logic  scheme.  The  number 
shown  In  each  box  represents  the  percentage  of  pilots  who  felt  that  Inhibition 
should  be  employed  for  that  alert  during  that  particular  flight  phase. 
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Figure  4.3.3. 1-1.  Alert  Prioritization  and  inhibit  Questionnaire  Rating  Form 
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Figure  4.3.3. 1  -2a.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts- 
Antiskid  L  Inbd  Fail 


Figure  4.3.3. 1  -2b.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts- APU  Fire 


- 

- 

-T 

* 

-1 

T 

♦ 

i 

_ j 

T 

♦ 

1 

_ J 

T 

* 

1 

r 

*-#-4 

_ 

_ 

T 

* 

1 

T 

* 

i 

1 

♦ 

1 

T 

* 

i 

T 

* 

1 

- 

T 


WARNING 

♦ 


I  L 


CAUTION 


1  f 


Figure  4.3.3. 1-3.  Means  and  95%  Confidence  Limits  for  Pilot  Ratings  of  Selected 
Caution  Level  Alerts 


Of  the  128  percentage  values  for  the  eight  flight  phases  shown  in  Table 
4. 3. 3. 2-1  only  23  (18%)  were  identified  as  requiring  inhibit  logic  by  a 
majority  of  the  pilots  questioned.  Overall,  there  was  little  agreement  as  to 
which  alerts  should  be  inhibited  during  each  flight  segment.  There  was  more 
agreement  as  to  when  alerts  should  not  be  inhibited;  in  several  cases  80  to 
90%  of  the  pilots  felt  that  inhibition  would  be  inappropriate.  Conversely, 
there  was  only  one  case  where  more  than  70%  of  the  pilots  agreed  on  the  need 
for  alert  Inhibition. 
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It  may  seem  surprising  that  approximately  50%  of  the  pilots  felt  that  a  fire 
warning  should  be  Inhibited  during  final  take  off  and  final  approach.  It 
should  be  noted  that  pilots  were  given  the  option  of  selecting  either  partial 
or  total  alerting  medium  Inhibition.  Inspection  of  the  raw  data  revealed  that 
90%  of  the  pilots  that  favored  Inhibiting  the  fire  alert  during  final  takeoff 
and  final  approach  preferred  that  only  one  component  of  the  fire  warning  be 
withheld.  Responding  pilots  suggested  that  they  would  want  the  fire  bell 
Inhibited  while  the  glareshield  light  would  remain  operational. 


Of  the  16  alerts,  there  were  only  4  for  which  one  third  or  more  of  the  pilots 
saw  a  need  for  some  type  of  aircraft  configuration  exception.  There  were,  on 
the  other  hand,  9  cases  where  80%  to  95%  of  the  pilots  saw  no  need  for  any 
configuration  exceptions. 


Table  4.3.3. 2-1.  Alert  Inhibit  Logic  Summary  * 


ALERTS 

/ 

FLIGHT  PHASE 

■i 

A  /  B  /  C  /  D  /  E  /  F  / 

/  /  /  /$  /  >/ 
,*/*/  $ 

yy 

^CONFIGUR- 

ATION 

y  EXCEPTIONS 

1.  ANTISKID  L  INBD  FAIL 

El 

19 

E3 

47 

42 

E3 

B 

El 

28 

2.  APUFIRE 

KJ3 

25 

E3 

25 

25 

E3 

El 

El 

33 

3.  L  FUEL  DUMP  VALVE  OPEN 

B 

29 

52 

43 

33 

33 

25 

35 

19 

4.  BATTERY  BUS  OFF 

« 

25 

40 

40 

□ 

E3 

35 

25 

25 

5.  GEN  OFF 

na 

15 

35 

25 

10 

□ 

40 

□ 

50 

6.  GALLEY  OVERHEAT 

19 

42 

57 

52 

23 

33 

42 

33 

5 

7.  APU  GEN  OFF 

25 

40 

60 

50 

35 

40 

55 

35 

20 

8.  CABIN  PRESS  RELIEF  VALVE  OPEN 

28 

47 

57 

47 

24 

33 

57 

45 

10 

9.  WING  ANTI-ICE  DISAGREE 

38 

47 

57 

43 

28 

38 

48 

48 

19 

10.  AIR  COND  PACK  OFF 

24 

43 

57 

57 

29 

43 

52 

43 

57 

1 1 .  OUCT  AVIONIC  COMP  OVERHEAT 

15 

35 

60 

50 

25 

35 

45 

□ 

5 

12.  ENG  FIRE 

14 

19 

47 

38 

14 

29 

52 

33 

33 

13.  L  EMER  AC  BUS  OFF 

25 

35 

a 

35 

20 

E3 

30 

30 

15 

14.  GPWS 

32 

42 

47 

42 

37 

32 

47 

52 

10 

15.  MANIFOLD  FAIL  (PNEU) 

23 

33 

57 

42 

23 

33 

47 

38 

10 

16.  CABIN  ALT 

45 

55 

66 

52 

14 

47 

71 

60 

10 

'Percentage  of  pilots  who  favored  alert  inhibition  during  each  flight  phase, 
and  percentage  who  foresaw  the  need  for  configuration  exceptions  for  each  alert. 


44  DISCUSSION 


44.1  DEBRIEFING  SUMMARY  RESULTS 

Host  of  the  data  collected  during  this  series  of  tests  supports  previous 
research  on  alerting  system  design.  This  Is  encouraging  when  It  Is  considered 
that  a  majority  of  the  pilots  Mho  participated  were  not  closely  associated 
with  research  such  as  this.  It  was  thought  that  many  of  the  line  pilots  who 
participated  would  see  Inconsistencies  between  what  they  observed  In  the 
simulator  and  what  they  actually  experience  while  flying  the  line.  Whenever  a 
number  of  advanced  concepts  are  demonstrated  Individually,  rather  than  as 
parts  of  a  cohesive  alerting  system,  pilots  sometimes  view  them  as  being 
somewhat  disjointed  and  confusing.  Fortunately,  the  opposite  was  true  for 
this  study.  The  participants  had  little  trouble  Integrating  these  concepts 
with  their  experiences  and  providing  some  valuable  Inputs  for  future  system 
development. 

There  was  no  clear  preference  for  either  the  split  or  separate  master  light, 
so  It  was  decided  that  the  split  ver  ion  of  the  master  visual  alert  would  be 
used  for  the  Phase  3  tests.  This  choice  was  consistent  with  the  conventional 
alerting  system  configuration  that  was  used  in  Phase  3.  This  configuration 
Is  used  In  2  of  the  3  major  wide  body  U.S.  aircraft. 

By  a  wide  margin,  the  pilots  preferred  a  master  light  In  conjunction  with  a 
flashing  box  for  annunciation  of  advisory  level  alerts.  It  was  agreed  that 
this  arrangement  would  be  employed  in  the  Phase  3  tests  along  with  a  discrete 
master  advisory  sound.  Available  guidelines  for  alerting  system  design 
recommend  the  use  of  a  distinct  sound  for  each  alert  priority  level  (Randle, 
Larsen  and  Williams,  I960;  Boucek,  Veltengruber  and  Smith,  1976;  MIL-STD-411D, 
1967;  and  MIL-STD-1472B,  1978).  During  high  levels  of  visual  workload,  a 
distinct  sound  will  provide  additional  sensory  Input,  and.  If  carefully 
chosen,  can  provide  extremely  effective  noise  penetrating  characteristics  as 
well  as  preliminary  Information  on  the  relative  severity  of  the  particular 
alert. 


Regarding  the  advisory  visual  information  display,  pilots  clearly  preferred  an 
alphanumeric  visual  display.  In  Phase  3,  advisory  level  alerts  were  presented 
on  the  same  visual  display  used  for  warning  and  caution  level  alerts. 

Although  a  majority  of  the  pilots  chose  a  combination  of  selective  and  total 
store/recall,  this  preference  was  not  significant.  The  store/recall  feature 
in  the  alerting  system  was  not  intended  to  be  evaluated  objectively  in  Phase 
3;  however,  a  demonstration  of  total  store/ recall  was  provided.  Accommodation 
of  voice  alerts  in  Phase  3  testing  was  provided  in  a  manner  consistent  with 
pilot  recommendations.  A  vast  majority  of  the  pilots  preferred  manual 
cancellation  of  voice  alerts  over  all  the  other  alternatives.  Manual 
cancellation  was  used  in  both  the  conventional  and  advanced  alerting  systems. 

In  addressing  the  subject  of  visual  message  syntax,  it  was  determined  that 
using  one  syntax  for  all  alert  messages  would  be  extremely  awkward.  Although 
a  significant  majority  of  the  pilots  favored  the  "heading-subsystem-nature  of 
problem"  configuration,  this  syntax  does  not  lend  itself  well  to  all  alert 
messages.  The  conclusion  drawn  is  that  standardization  should  be  the  goal  but 
a  clear  statement  of  the  problem  is  an  imperative  prerequisite. 

The  issue  of  verbal  alert  generated  some  rather  interesting  results.  Most  of 
the  pilots  (though  not  a  significant  majority)  favored  voices  over  discrete 
tones.  This  was  particularly  surprising  since  a  majority  of  the  pilots  had 
never  been  exposed  to  this  concept  before.  As  mentioned  earlier,  the 
available  literature  suggests  that  a  precursor  tone  is  effective  in  its 
attention-getting  capability.  For  this  reason,  the  precursor  tones  were  used 
in  the  Phase  3  testing.  However,  it  is  hoped  that  the  unexpected  pilot 
acceptance  of  master  voice  alerts  will  encourage  research  efforts  aimed  at 
measuring  the  relative  effectiveness  of  these  two  alerting  concepts. 

Regarding  multiple  verbal  alerts,  most  of  the  pilots  questioned  favored 
introduction  of  the  message  "Multiple  Alerts"  in  cases  where  2  or  more  alerts 
occur  simultaneously.  This  option  appeared  to  oe  the  most  viable  in  terms  of 
facilitating  crew  awareness  of  the  situation  as  well  as  being  conducive  to 
expedient  corrective  action.  Hie  other  two  options  required  a  prioritization 
scheme  which  Is  not  presently  available  In  an  accurate  and  reliable  form.  For 
these  reasons,  the  option  preferred  by  a  majority  of  pilots  was  implemented 
for  Phase  3  activity. 


46 


4.4.2  SUPPLEMENTAL  DEBRIEFING  QUESTIONNAIRE 

Responses  to  the  supplemental  debriefing  questionnaire  Indicate  that,  for  the 
most  part,  pilots  were  satisfied  with  the  preliminary  development  of  the 
central  display  unit.  The  overall  trend  was  toward  a  preference  for  the 
display  format  employing  reverse  chronology  within  alert  urgency  level.  This 
format  was  Incorporated  Into  the  Phase  3  testing. 

4.4.3  ALERT  PRIORITIZATION  AND  INHIBIT  QUESTIONNAIRE 

There  are  a  number  of  conclusions  to  be  drawn  from  the  results  of  the  alert 
prioritization  and  Inhibit  questionnaire.  It  seems  likely  that  prioritization 
and  inhibit  systems  need  to  be  aircraft  and  possibly  airline  specific  as  there 
was  wide  variability  of  ratings  among  pilots  for  the  16  alerts  used  in  the 
questionnaire.  There  may  be  two  explanations  for  this  situation.  The  first 
is  that  the  pilots  who  participated  in  this  study  represented  quite  a  variety 
of  experience  and  backgrounds  which  was  generally  a  function  of  the  airline 
and  aircraft  types  flown.  The  number  of  aircraft  types  represented,  may  have 
caused  the  large  amount  of  the  observed  variability.  Secondly,  there  may  have 
been  a  certain  amount  of  disagreement  among  pilots  as  to  the  relative 
importance  of  various  alerts  that  was  not  due  to  specific  aircraft  experience. 
Even  with  subject  variability  taken  into  account,  the  data  suggests  the  need 
for  a  prioritization  system  that  is  flight  phase  adaptive.  This  is  consistent 
with  recommendations  made  by  other  researchers  (Randle,  Larsen  and  Williams, 
1980). 

In  developing  an  effective,  reliable  prioritization  system,  a  more  precise 
methodology  is  needed.  In  looking  at  the  graphic  representations  of  the  pilot 
responses,  it  is  clear  that  construction  of  a  useful  prioritization  scheme 
would  be  impossible  using  this  data.  It  may  be  that  other  organizations  such 
as  Reliability  and  Safety  could  provide  precise  data  that  would  be  used  to 
generate  an  effective  prioritization  system. 

There  was  also  very  little  agreement  among  pilots  as  to  which  alerts  should  be 
inhibited  during  each  flight  phase.  Although  a  good  percentage  of  pilots 
favored  inhibition  of  relatively  serious  fault  messages  during  the  final 
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takeoff  and  final  approach  flight  segments,  90%  of  those  who  felt  this  way, 
however,  favored  Inhibition  of  only  one  component  of  the  alert. 

Alert  prioritization  and  Inhibit  systems  are  In  an  early  stage  of  development. 
Additional  research  should  be  conducted  on  subjects  such  as  alert  sequencing. 
Inhibit  justification  and  requirements  for  aircraft  specific  systems. 


5.0  SYSTEM  SIMULATION  TESTS 

The  primary  purpose  of  the  Phase  3  testing  effort,  as  originally  defined,  was 
to  evaluate  the  candidate  alerting  concepts  developed  in  Phase  1,  and  to 
identify  those  candidates  which  were  valid  alternatives  to  conventional 
alerting  techniques.  These  candidate  concepts  were  then  to  be  translated  into 
alerting  system  design  guidelines.  Due  to  the  high  level  of  agreement 
achieved  among  the  working  committee  members  concerning  the  conceptual  system, 
the  number  of  alternative  alerting  concepts  was  smaller  than  anticipated. 

This  fact,  combined  with  the  identification  of  two  other  major  areas  of 
interest,  the  flight  engineer's  station  and  time-critical  warnings,  resulted 
in  a  modification  of  the  original  simulation  test  program. 

The  system  validation  portion  of  the  simulation  program  was  conducted  as 
planned;  pilot  performance  using  the  two  alternative  advanced  crew  alerting 
systems  was  evaluated  in  comparison  to  a  conventional  system.  The  flight 
engineer's  performance  using  an  advanced  display  was  also  compared  to  a 
conventional  system  to  validate  the  advanced  concept.  Finally,  a  test  was 
designed  to  evaluate  various  presentation  media,  formats  and  locations  for 
those  warnings  which  have  highly  time-critical  response  requirements,  such  as, 
ground  proximity,  collision  avoidance,  windshear.  Each  of  the  objective 
tests,  supplemented  with  a  questionnaire  to  obtain  pilot  opinions  was  analyzed 
and  the  findings  were  incorporated  into  the  alerting  system  guidelines.  The 
following  sections  describe  these  tests,  conducted  in  the  Boeing  Visual  Flight 
Simulator,  and  summarize  the  results. 

5.1  TEST  OBJECTIVES 

5.1.1  SYSTEM  VALIDATION 

The  first  objective  of  the  Phase  3  tests  was  to  evaluate  the  candidate 
alerting  system  concepts  to  determine  their  validity  with  respect  to:  1)  the 
objectives  of  a  crew  alerting  system,  2)  the  assumptions  upon  which  the 
systems  were  based,  3)  the  functioning  of  the  system  components  in  a  simulated 
real-world  setting,  and  4)  the  acceptability  of  the  system  by  the  users. 
Therefore,  tests  and  evaluations  were  designed  to  answer  the  following 
questions  concerning  the  validity  of  the  candidate  alerting  concepts. 
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1.  Do  the  candidate  concepts  meet  the  assumptions  set  forth  concerning 
advanced  systems  (see  Table  1.0-1)? 

2.  Do  the  candidate  concepts  meet  the  objectives  of  an  alerting  system  (see 
Section  1.0)? 

3.  Do  the  concepts  enable  equal  or  better  pilot  detection  and  response 
performance  than  a  conventional  alerting  system? 

4.  Is  the  information  provided  by  the  advanced  systems  equal  to  or  better 
than  that  provided  by  a  conventional  alerting  system? 

5.  Is  the  flight  engineer's  performance  with  the  advanced  system  equal  to  or 
better  than  performance  with  a  conventional  system? 

6.  Are  the  detection  and  response  times  for  the  different  urgency  levels 
affected  by  the  system  used? 

7.  How  often  is  voice  used  in  candidate  system  B?  How  can  voice  best  be 
used  in  ad.anced  alerting  systems? 

8.  How  do  the  pilots  who  fly  with  the  advanced  systems  rate  them  with 
respect  to  conventional  systems? 

5.1.2  TIME-CRITICAL  WARNING 

A  second  objective  in  the  Phase  3  tests  was  to  define  alerting  methods  which 

would  enable  the  pilot  to  respond  quickly  and  accurately  to  situations  which 

are  extremely  time-critical.  With  this  objective,  tests  and  evaluations  were 

designed  to  answer  the  following  questions: 

1.  Is  a  time-critical  display  needed? 

2.  Does  the  location  of  the  time-critical  display  have  an  effect  on 
detection  time,  response  time,  and/or  missed  alerts? 


3.  Is  there  any  difference  in  response  performance  when  the  alerts  are 
presented  graphically,  alphanumerical ly  or  by  a  combination  of  the  two? 

4.  Do  responses  change  as  a  function  of  different  combinations  of  display 
location  and  presentation  formats? 

5.  Is  there  an  effect  on  response  performance  when  the  mes::.;^e  format 
provides  the  pilot  with  guidance  (e.g.,  PULL  UP),  rather  tiian  status 
information  (e.g.,  TERRAIN)? 

6.  What  is  the  effect  on  detection  and  response  performance  with  the 
presence  or  absence  of  the  voice  message? 

7.  Do  time-critical  warnings  have  a  disruptive  effect  on  flight  performance? 
5.2  EXPERIMENTAL  DESIGN 

The  basic  experimental  design  was  a  factorial  analysis  of  variance  with 
repeated  measures.  The  following  sections  present  the  specific  designs  for 
each  test  and  the  variables  chosen  for  study. 

5.2.1  SYSTEM  VALIDATION  TESTS 

The  validation  tests  compared  pilot  performance  on  the  advanced  and 
conventional  systems.  There  were  three  validation  tests.  Each  of  two 
advanced  alerting  pilot  systems  were  compared  to  a  conventional  system;  an 
advanced  flight  engineer  system  was  compared  to  a  conventional  system.  Each 
of  the  pilots  used  one  of  the  advanced  systems  for  two  flights  and  the 
conventional  system  for  two  flights.  Half  of  the  pilots  flew  concept  A  and 
half  concept  B,  to  negate  possible  confounding  effects. 

Figure  5. 2. 1-1  illustrates  the  arrangement  of  the  alerting  components  for  the 
candidate  systems.  Each  system  consisted  of  three  components.  The  first  was 
a  split  legend  master  visual  alert  located  in  the  pilot's  primary  field  of 
vision.  The  upper  half  of  the  master  alert  was  red  and  labelled  WARNING;  the 
lower  half  was  amber  and  labelled  CAUTION.  The  second  component  was  the 


MASTER  AURAL  ALERTS  AND 
VOICE  INFORMATION  DISPLAY 


WARNING 

CAUTION 

ADVISORY 


(R)  Red 

(A)  Amber 

(B)  Blue 

Figure  5.2. 1-2.  Typical  Format  for  the  Advanced  Information  Display 


CABIN  ALTITUDE  (R) 

UPPER  YD  FAIL  (A) 

L  ENG  OIL  PRESS  (A) 

CAT  III  FAULT  (B> 


programmable  information  display.  Figure  5.2. 1-2  presents  a  typical  format 
for  the  messages.  The  alphanumeric  display  used  in  the  test  was  capable  of 
presenting  12  lines  of  messages  with  16  characters  per  line.  Character  size 
was  0.2  inch  high  by  0.1  inch  wide;  character  separation  was  0.08  inch,  word 
separation  was  0.26  inch  or  one  letter  space;  and  line  separation  was  0.05 
inch.  Warning  messages  were  presented  in  red,  cautions  in  amber,  and  advis¬ 
ories  in  blue.  Each  new  alert  message  was  surrounded  by  a  flashing  box  of  the 
same  color  to  indicate  that  it  was  a  new  alert.  The  flash  rate  of  the  box  was 
four  times  per  second  with  equal  times  for  "on”  and  "off".  Since  the  Phase  1 
testing  had  investigated  the  presentation  of  alerts  with  other  messages 
already  on  the  display,  a  duplication  of  that  was  not  necessary;  and  the  alert 
messages  were  presented  on  a  blank  display.  Finally,  a  dedicated  speaker  was 
used  for  the  master  aural  alerts  and  the  voice  messages.  This  speaker  was 
located  to  the  left  of  the  pilot  and  separated  by  approximately  90°  from  the 
speaker  used  to  present  ATC  communication.  The  sounds  used  for  the  master 
aural  alerts  were  selected  using  two  criteria.  First,  the  sound 
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had  to  be  unique  and  easily  distinguishable;  second,  they  should  not  convey 
any  stereotypical  meaning  to  the  pilot,  (e.g.,  bell  =  fire).  To  meet  these 
requirements  the  following  sounds  were  chosen: 

Warning  -  A  sound  characteri zed  as  a  European  police  car  siren.  This  sound 

consisted  of  two  tones  (high,  660Hz  and  low,  330Hz)  which  alternated 
back  and  forth  at  a  rate  of  two  times  a  second.  The  shape  of  the 
sound  is  given  in  Figure  5.2. 1-3. 

Caution  -  A  short  pulsing  sound.  The  original  intention  was  to  choose  a 

steady-state  sound  for  caution  alerts.  It  was  determined,  however, 
that  the  number  of  steady  state  sounds  in  present-day  use  was  so 
large  that  the  requirement  for  avoiding  stereotypic  sounds  could  not 
be  met;  therefore,  a  unique  sound  was  developed  for  testing.  The 
caution  sound  started  at  the  low  tone  (200Hz)  with  a  60  msec  rise  to 
the  high,  (800Hz)  a  40  msec  presentation  of  the  high,  a  60  msec  fall 
to  the  low,  and  a  40  msec  silent  time.  Four  such  cycles  were 
presented  for  the  caution  alert.  The  shape  of  the  caution  sound  is 
shown  in  Figure  5.2. 1-3. 

Advisory-  Single  stroke  chime.  A  475Hz  tone  was  presented  for  2.0  sec  with  a 
50  msec  rise  and  a  1.8  sec  fall  in  intensity.  The  shape  of  the 
sound  is  shown  in  Figure  5.2. 1-3. 

The  peak  intensity  levels  for  the  tones  were  adjusted  to  approximately  78  dB 
which  was  8  dB  above  the  average  ambient  noise  in  the  simulator  flight  deck. 
Voice  messages  that  occurred  automatically  (system  A)  were  preceded  by  a  0,75 
seconds  presentation  of  the  master  warning/caution  aural  alerts.  The  voice 
message  was  then  repeated  until  the  pilot  made  the  correct  response  or 
cancelled  the  alert.  The  off- time  interval  between  repetitions  was  0.2 
seconds.  Peak  intensity  levels  for  the  voice  were  also  held  at  approximately 
78  dB.  The  voice  messages  for  system  B  were  not  presented  automatically  but 
rather  by  pilot  action.  For  this  system,  the  warning  and  caution  master  aural 
alerts  continued  until  the  pilot  selected  the  voice  message,  cancelled  the 
aural  alert  manually  or  solved  the  problem.  The  voice  message  was  presented 
only  when  the  pilot  initiated  an  action  to  hear  it,  and  the  message  was 
presented  once  for  each  selection. 
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Figure  5.2. 1  -3.  Master  Alerting  Sounds 


The  flight  engineer's  station  was  also  equipped  with  a  programmable 
information  display  which  presented  the  same  messages  that  appeared  on  the 
pilot's  display.  The  flight  engineer  also  heard  the  same  aural  alerts  as  the 
pilot  but  did  not  have  the  ability  to  manually  cancel  them  or  to  call  for  a 
repeat  of  voice  messages.  The  location  of  the  flight  engineer's  information 
display  is  shown  in  Figure  5. 2. 1-4. 

Figure  5.2. 1-5  illustrates  the  arrangement  of  the  alerts  in  the  conventional 
pilot  system.  As  stated  in  Section  1.2,  this  baseline  system  was  not  designed 
to  represent  any  specific  existing  aircraft  but  rather  to  be  representative  of 
those  in  present  use.  The  warnings  were  presented  by  discrete  annunciators 
(as  can  be  seen  from  the  position  of  the  FIRE  and  GEAR  lights  in  Figure 
5. 2. 1-5);  they  were  also  accompanied  by  discrete  tones  (such  as  for  CABIN 
ALTITUDE  and  OVERSPEED).  All  cautions  illuminated  the  master  caution  light 
plus  either  a  distributed  light,  (e.g.,  ALTITUDE  or  LE  FLAPS)  or  a  light  on 
the  annunciator  panel.  The  ALTITUDE  alert  was  also  accompanied  by  a  discrete 
tone.  All  advisories  illuminated  lights  on  the  annunciator  panel. 

The  discrete  aural  alerts  were  presented  at  an  intensity  of  approximately  85 
dB  or  the  prescribed  15  dB  above  ambient  noise. 

The  flight  engineer's  conventional  alerting  system  also  consisted  of 
distributed  alerts.  Since  the  flight  engineers  station  has  no  direct  warning 
indicators,  warnings  were  interpreted  from  the  pilot’s  annunciations.  Cautions 
and  advisories  were  presented  as  lights  on  the  active  panels  or  on  the 
annunciator  panel.  Figure  5. 2. 1-6  illustrates  the  location  of  the  active 
system  panels  and  the  annunciator  panel.  The  programmable  information  display 
was  not  operational  for  the  conventional  alerting  system. 

5.2.2  TIME-CRITICAL  WARNING  TEST 

The  basic  test  configuration  for  the  time-critical  warning  test  is  shown  in 
Figure  5. 2. 2-1.  There  were  four  independent  variables  for  this  test:  a) 
display  location,  b)  presentation  format,  c)  voice  format,  and  d)  message 
content.  The  display  location  variable  had  two  levels:  within  the  pilots 
primary  field  of  view  (15°)  or  within  the  pilot's  secondary  field  of  view 
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Figure  5.2. 1-4.  Flight  Engineer  Advanced  Warning  and  Caution  Display 
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•  Secondary  tasks 

Figure  5.2.2-1.  Test  Configuration  For  The  Time-Critical  Warning  Tests 


(30°).  The  presentation  format  variable  had  three  levels:  al phanuneric 
messages,  graphic  messages  and  a  combination  of  alphanumeric  and  graphic 
messages.  The  voice  format  variable  had  two  levels:  the  presence  or  the 
absence  of  voice  alerts.  The  message  content  variable  had  two  levels:  the 
messages  provided  the  pilot  with  status  information,  (e.g.,  TERRAIN),  or  gave 
the  pilot  guidance  as  to  the  correct  action,  (e.g.,  PULL  UP).  Thus,  the  24 
cells  shown  in  Figure  5. 2.2-1  represent  a  2  x  3  x  2  x  2  factorial  analysis  of 
variance  design.  A  mixed  model  design  was  used  with  repeated  measures  on  the 
first  three  variables  since  all  subjects  received  all  treatments.  However, 
the  subjects  were  nested  within  the  levels  of  message  content  since  each  had 
only  one  content.  Figure  5. 2.2-2  illustrates  the  locations  of  the  information 
display  in  the  central  panel  and  the  display  located  in  front  of  the  pilot 


Figure  5. 2. 2-2.  Time  Critical  Display  Location 


(30°  field  of  view  and  15°  field  of  view,  respectively).  The  ffgure  also 
shows  the  location  of  the  master  visual  alert  with  respect  to  the  two 
progranmable  display  locations.  The  top  1.5  Inches  of  each  of  prograimnabl e 
display  was  used  to  present  time-critical  warnings.  The  bottom  3  Inches  of 
the  centrally  located  display  was  used  as  the  alerting  system  Information 
display  and  presented  all  alerts.  The  center  display  was  therefore  capable  of 
presenting  the  time-critical  warnings  and  eight  lines  of  alphanumeric 
information  with  16  characters  per  line.  Figure  5. 2. 2-3  presents  a  typical 
display  format.  The  display  In  front  of  the  pilot  only  presented 
time-critical  warnings. 
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•  16  characters  per  line 


Figure  5.2.2-3.  Typical  Time-Critical  Test  Display  Format 


62 


r 


To  evaluate  the  media  for  presenting  time-critical  warnings  It  was  necessary 
to  compare  them  only  with  other  warnings;  therefore,  all  alerts  In  this  test, 
except  for  multiple  alerts,  were  warnings  (red).  Since  the  Phase  1  testing 
Investigated  the  presentation  of  alerts  with  other  messages  already  on  the 
screen,  a  duplication  of  that  test  was  not  necessary,  and  the  alert  messages 
appeared  on  a  blank  screen. 

Both  the  master  aural  alerts  and  the  voice  messages  were  presented  on  a 
speaker  located  to  the  left  of  the  pilot.  The  intensity  of  the  alerts  was  set 
at  78dB  or  approximately  8dB  above  the  average  ambient  noise. 

The  rationale  for  selecting  the  display  locations,  presentation  format,  voice 
format  and  message  format  variables  was  as  follows: 

All  variables  were  chosen  to  satisfy  the  underlying  requirement  to  elicit  a 
quick  and  accurate  response  from  the  pilot.  Each  variable  had  as  a  basis  some 
facet  of  the  advanced  alerting  concepts.  The  centralized  display  location 
within  the  pilot's  secondary  field  of  view  (within  30°  of  the  centerline  of 
vision)  was  shown  in  Phase  1  to  be  adequate  when  combined  with  a  master 
attention-getting  alert.  However,  moving  the  information  into  the  primary 
field  of  view  (15°)  for  a  certain  set  of  warnings  decreased  the  pilot’s 
response  time  to  those  messages. 

The  second  variable  investigated  was  the  presentation  format.  The 
programmable  information  display  presented  the  pilot  with  alphanumeric  alert 
messages.  This  required  the  pilot  to  read  the  message,  understand  its 
content,  decide  a  course  of  action  and,  finally,  respond.  This  sequence  and 
the  time  it  takes  is  appropriate  for  most  alerts.  However,  if  time  is  a 
critical  factor  it  may  be  possible  to  reduce  the  overall  time  from  alert  onset 
to  pilot  response  by  providing  a  graphic  representation  of  the  alert  to 
facilitate  interpretation,  or  by  combining  graphic  with  alphanunerlc 
information.  In  conjunction  with  the  presentation  format,  the  third  variable 
message  content  may  also  be  used  to  reduce  the  time  from  alert  to  response. 

The  normal  method  of  alert  presentation  is  to  give  the  pilot  the  status  of  his 
aircraft,  e.g.,  "LEFT  ENGINE  FIRE"  or  "OVERSPEED".  This  requires  the  pilot  to 
follow  the  above  sequence  remembering  the  appropriate  response  and  executing 
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it  correctly.  An  alternative  content  for  time-critical  messages  would  be  to 
give  the  pilot  guidance  by  providing  the  correct  course  of  action  which  should 
accelerate  the  response  and  decrease  the  probability  of  error.  To  eliminate 
the  possibility  of  negative  transfer  between  the  status  and  guidance  message 
formats,  half  of  the  pilots  were  given  status  messages  and  half,  guidance. 

The  alphanumeric  and  graphic  presentation  used  for  both  status  and  guidance 
messages  are  shown  in  Figure  5. 2.2-4. 

The  final  variable  investigated  was  the  value  of  the  voice  message.  Although 
it  has  been  shown  that  voice  messages  have  the  potential  to  significantly 
interfere  with  other  voice  communications  in  the  flight  deck,  the  severity  and 
time-criticality  of  some  warnings  may  make  a  chance  of  voice  alert 
interference  appropriate.  The  question  remains  whether  or  not  the  voice  alert 
changes  crew  response  time  or  performance  accuracy  in  any  way. 

5.3  PILOT  SAMPLE  FOR  THE  SYSTEM  TESTS 

Fourteen  pilots  with  a  wide  range  of  experience,  including  line  pilots, 
instructors,  and  management  pilots,  participated  in  the  Phase  3  tests  at  the 
Boeing  facility.  The  group  consisted  of  representatives  from  the  three 
contractor  companies,  Boeing,  Lockheed  and  McDonnell  Douglas,  and  from  the 
airlines  including  American,  Eastern,  Northwest,  SAS,  TWA,  United,  and 
Western.  A  summary  of  the  pilot  experience  is  presented  in  Table  5.3-1; 
nunerical  entries  on  the  right  hand  side  of  the  table  indicate  the  specific 
experience  by  aircraft  type  and  recency  of  the  experience  (A  is  most  recent). 

5.4  CREW  TASKS 

5.4.1  FLIGHT  TASK 

To  simulate  the  flight  deck  environment  and  work  pattern,  the  pilots  performed 
test  flights  of  31  minutes  duration  in  the  simulator.  A  realistic  aircraft 
model  was  used  for  the  basic  flying  task;  the  pilots  were  required  to  fly  a 
prescribed  flight  plan,  respond  to  ATC  communications,  locate  targets  in  an 
external  visual  scene  and  to  respond  to  alerts.  The  flight  instrumentation 
available  to  the  pilots  to  perform  their  tasks,  shown  in 
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Guidance 


Figure  5.2.2  4.  Time-Critical  Warning  Display  Formats 
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3S.0 

D 
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*A  is  the  most  recent  aircraft  flown. 


Table  5.3-1.  Summary  of  Pilot  Experience 


Figure  5.4. 1-1,  consisted  of  an  airspeed  indicator;  an  electronic  altitude 
direction  indicator  (EADI-roll,  pitch,  glideslope);  an  altimeter;  a  rate  of 
climb  indicator;  a  horizontal  situation  indicator  (HSI-course,  DME, 
localizer);  the  pilot's  time-critical  display;  and  a  clock  to  indicate  flight 
time.  The  center  panel  contained  the  visual  information  display,  the 
electronic  engine  instrument  display,  flaps  indicator  and  gear  lights. 

The  flight  controls  available  to  the  pilot  included:  wheel  and  column  with 
trim;  rudder  and  toe  brakes;  speed  brakes;  flap  handle;  gear  handle;  fire 
handles;  throttle;  response  key  matrix  and  a  12  key  input  panel. 

The  test  flight  plan  is  illustrated  in  Figure  5. 4. 1-2.  It  was  divided  into 
five  flight  phases,  takeoff,  climb,  cruise,  descent  and  landing.  The  pilot 
performed  a  visual  takeoff  (Figure  5. 4. 1-3)  on  a  heading  of  360°  at  a  rate  of 
climb  resulting  from  IAS  of  210  Kts.  The  outside  visual  scene  disappeared 
after  takeoff.  To  achieve  a  more  controlled  flight  path  for  the  flights,  the 
auto  throttle  was  engaged  at  2000  feet  and  flew  the  prescribed  speed  profile 
for  the  remainder  of  the  flight.  The  pilot  leveled  off  and  held  15000  feet 
through  turns  1,  2  and  3.  At  a  point  10  miles  from  waypoint  D  he  received  an 
ATC  clearance  to  descend  to  10,000  feet.  After  executing  turn  4  ATC  cleared 
the  aircraft  to  4000  feet  and  instructed  the  pilot  to  hold  that  altitude  until 
crossing  the  mountain  9.5  miles  from  the  runway.  At  18  miles  out,  the  outside 
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Figure  5.4. 1-3.  Visual  Flight  Takeoff 


visual  scene  again  became  active  for  the  approach.  At  9.5  miles  he  was 
further  cleared  for  ILS  approach  and  landing.  The  glideslope  raw  data  box 
appeared  on  the  EADI,  and  the  visual  approach  continued,  until  3  miles  and 
1000  feet  at  which  point  the  visual  scene  again  disappeared  to  enable  the 
outside  camera  to  switch  from  one  terrain  board  to  another.  At  one  mile  and 
350  feet  the  visual  scene  was  again  presented  for  landing.  The  ATC  clearances 
associated  with  the  flight  plan  are  presented  in  Table  5.4. 1-1. 

5.4.2  TARGET  RECOGNITION 

To  assure  that  the  pilots  spent  some  of  their  time  looking  outside  the  cab  at 
times  other  than  takeoff  and  landing,  an  outside  visual  task  was  developed. 
Occasionally  during  each  flight,  ATC  would  request  the  pilot  to  report  traffic 
location  (Boeing  101,  say  current  traffic  location).  The  traffic  was 
simulated  by  using  groups  of  split  rings  all  of  which  were  oriented  in  the 
same  direction  except  for  one  target  ring  (see  Figure  5. 4. 2-1).  The  pilot's 
task  was  to  search  three  groups  of  rings,  find  the  single  target  ring,  and 
report  its  position.  Reports  were  made  by  inputting  the  target  location  in  a 
prescribed  manner  through  the  12-key  keyboard.  This  task  was  reported  by  some 
of  the  pilots  to  be  very  comparable  to  an  actual  in-flight  traffic  search 
except  that  in  actual  flight  ATC  would  normally  give  them  some  idea  where  the 
traffic  was,  e.g.,  "Boeing  101  you  have  traffic  at  two  o'clock".  Therefore, 
the  test  task  was  possibly  a  little  more  difficult  than  real  world  traffic 
identification.  Figure  5. 4. 2-2  shows  the  correct  target  screen/quadrant 
number  for  each  slide  set  used  in  the  study. 

5.4.3  FLIGHT  ENGINEER'S  FLIGHT  TASK 

The  work  pattern  and  task  loading  of  the  flight  engineer  was  very  difficult  to 
simulate,  the  flight  engineer  generally  has  a  higher  loading  at  the  beginning 
and  end  of  each  flight  and  is  somewhat  unloaded  during  the  middle  portion. 
However,  if  a  problem  occurs  there  may  be  a  dramatic  increase  in  workload.  To 
simulate  worst-case,  or  near  worst-case,  the  workload  for  the  test  flight  was 
artificially  high. 
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Table  5.4. 1-1 .  A  TC  Communication 


Distance 

Altitude 

ATC 

1 

0 

0 

Boeing  101:  Pinevalley  Tower:  cleared  for  takeoff 
runway  36,  wind  calm  altimeter  29.92'.  Cleared 
left  heading  310  deg  at  fix  ALPHA,  Monitor 
Pinevalley  Approach  Control  348.2  after  takeoff. 

2 

8  nmi 

9,000 

Boeing  101:  Pinevalley  Approach  Control:  say 
current  traffic  location. 

3 

21.7  nmi 

15,000 

Boeing  101:  Pinevalley  Approach  Control:  fix 
BRAVO  turn  left  heading  242  maintain  15,000. 

4 

29.3  nmi 

15,000 

Boeing  101:  Pinevalley  Approach  Control:  fix 
COCOA,  turn  left  heading  169  maintain  15,000. 

5 

44.3  nmi 

15,000 

Boeing  101:  Pinevalley  Approach  Control:  say 
current  traffic  location 

6 

56.3  nmi 

15,000 

Boeing  101:  Pinevalley  Approach  Control:  say 
current  traffic  location 

7 

71  nmi 

15,000 

Boeing  101:  Pinevalley  Approach  Control: 
descend  to  10,000,  cleared  penetration  and 

1 LS  approach  runway  36 

8 

72.5  nmi 

14,300 

Boeing  101:  Pinevalley  Approach  Control:  say 
current  traffic  location 

9 

90.8  nmi 

10,000 

Boeing  101:  Pinevalley  Approach  Control:  have 
you  starting  approach,  do  not  descent  below 

4,000  feet  until  DME  9.5  nmi,  current  winds 
light  and  variable  altimeter  29.92,  monitor 
Pinevalley  Tower  253.8 

10 

112.3  nmi 

4,000 

Boeing  101:  Pinevalley  Tower:  cleared  to  land 
runway  36 

O  Q  °  o  ° 
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Figure  5.4. 2-2.  Answer  Sheet  for  Slide  Sets 
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The  flight  engineer  flew  at  the  same  time  as  a  pilot.  Their  tasks  consisted 
of  reading  instruments,  logging  problems  or  faults  and  their  time  of 
occurrence,  and  locating  the  outside  visual  targets.  The  instrumentation  that 
was  functional  at  the  flight  engineer's  station  is  illustrated  in  Figures 
5.4. 3-1,  and  -2.  The  electrical  and  hydraulic  systems  had  round  dial 
instruments,  and  the  three  round  dial  indicators  on  the  hydraulic  system  panel 
were  switchable  to  provide  readings  for  each  of  the  three  hydraulic  systems 
(left,  center  and  right);  they  therefore  represented  9  dial  instruments.  The 
electrical  and  fuel  instruments  were  not  switchable. 

At  certain  predetermined  intervals  (10  times  during  each  flight),  the  reading 
on  one  of  the  instruments  would  drop;  the  flight  engineer's  task  was  to  detect 
the  low  reading  and  log  the  time  of  occurrence.  The  task  required  the  flight 
engineer  to  scan  the  instruments  during  the  flight  as  well  as  to  search  for 
and  identify  the  outside  visual  targets.  The  flight  engineers  were  also 
required  to  log  all  system  alerts  after  they  had  responded  to  them. 

5.4.4  ALERT  RESPONSE  TASKS 

When  the  pilots  detected  an  alert,  they  were  required  to  depress  a  button 
located  on  the  left  side  of  the  control  wheel.  This  action  was  used  to  mark 
the  time  that  the  pilot  perceived  the  new  alert.  After  identifying  the 
specific  alert,  the  pilot  performed  a  prescribed  response  to  solve  the 
problem.  Table  5. 4. 4-1  presents  the  operational  or  system  conditions  that 
were  used,  along  with  their  associated  responses.  As  can  be  seen,  the 
responses  were  divided  between  two  categories,  those  that  were  made  with 
operable  system  elements  (e.g.,  wheel  back,  cycle  gear,  etc)  and  those  that 
were  made  through  a  response  panel  by  depressing  the  switch  corresponding  to 
the  system  which  had  a  problem  (e.g.,  L  SYS  HYD  PRSR,  ANTI-ICE).  The  response 
panel  had  18  switches  located  in  the  center  aisle  stand,  and  configured  as 
seen  in  Figure  5. 4. 4-1.  Caution  and  advisory  level  alerts  were  always 
responded  to  through  this  panel.  When  the  pilot  made  the  correct  response, 
the  alert  message  was  removed  from  the  screen,  the  master  visual  alert  was 
extinguished  and  the  aural  alerts  were  silenced. 
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Figure  5.4.3 - 1.  Flight  Engineer's  Lower  Panel 
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Figure  5.4.4- 1  Pilot's  Response  Panel 


Figure  5.4.4-2  Flight  Engineer's  Response  Panel 


The  flight  engineers'  response  to  alerts  was  not  the  same  as  the  pilots  as 
they  were  not  required  to  signal  detection  before  responding.  Their  responses 
were  also  divided  into  two  categories  based  on  operable  systems  but  the 
distinction  was  not  as  clear  cut.  Warnings  always  required  a  response  panel 
input  (see  Figure  5. 4. 4-2  for  the  flight  engineer's  response  panel),  while 
caution  and  advisory  level  alerts  could  be  responded  to  either  on  the 
appropriate  system  panel  or  on  the  response  panel  depending  on  the  specific 
alert.  Table  5. 4. 4-1  presents  the  flight  engineer  response  for  each  alert. 

5.5  TEST  PROCEDURES 

5.5.1  SYSTEM  VALIDATION  TESTS 

The  variables  tested  in  the  svstem  validation  tests  are  described  in  section 
5.2.1.  All  variables  not  tested  were  held  constant  or  controlled  to  avoid 
biasing  or  confounding  the  results.  Simulated  aircraft  ambient  noise  r^t.h  an 
average  intensity  of  approximately  70  dB  was  presented  during  the  flight  task 
to  mask  the  uncontrolled  noise  that  may  have  been  occurring  around  the  cab. 
This  noise  was  controlled  by  throttle  position  and  air  speed  to  provide  a 
realistic  sound  spectrum  based  on  aircraft  performance.  During  each  flight, 
variations  of  the  noise  level  were  kept  within  the  range  of  67dB  to  72dB.  The 
ambient  light  levels  were  kept  very  low  (5  ft-L)  to  permit  the  use  of  the 
outside  visual  scene.  ATC  communications  were  presented  at  75dB  and  held 
constant  for  all  trials;  visual  message  constrast  was  also  held  constant  for 
all  trials.  All  pilots  received  the  same  instructions  to  minimize 
experimenter  bias  (see  Appendix  F). 


Table  5.4.4-1.  Operational  and  System  Conditions  for  Alerts  and  Their  Associated  Response 


Alert 

Alert 

code 

CRT  message 

Urgency 
level  " 

Pilot's  response 

Flight 

engineer's 

response 

Ground  proximity 

1 

GROUND  PROX 

W 

COLUMN  BACK 

- 

Thrust  reserves  too  low 

THR  RSV  TOO  LOW 

W 

THROTTLE  FORWARD 

- 

Collision  avoidance 

COLLISN  ABOVE 

W 

COLUMN  FORWARD 

- 

Collision  avoidance 

COLLISN  LOW  LEFT 

w 

COLUMN  BACK 

WHEEL  RIGHT 

- 

Tailwind  shear 

TAILWIND  SHR 

w 

THROTTLE  FORWARD 

- 

Downdraft  shear 

DOWNDRAFT SHR 

w 

THROTTLE  FORWARD 

- 

Left  engine  fire 

8 

LENG  FIRE 

w 

PULL  LEFT  FIRE  HANDLE 

RP  FIRE* 

APU  fire 

9 

APU  FIRE 

w 

PULL  CENTER  FIRE  HANDLE 

RP  FIRE 

Flaps  set  improperly 

10 

TAKEOFF  FLAPS 

w 

CYCLE  FLAP  HANDLE 

RP  CONFIG 

Flaps  set  improperly 

15 

LANDING  FLAPS 

w 

CYCLE  FLAP  HANDLE 

RP  CONFIG 

Right  engine  failure 

11 

R  ENG  FAIL 

w 

RP  ENG  STATUS 

RP  ENG  STATUS 

Gear  not  down 

12 

GEAR  NOT  DOWN 

w 

CYCLE  GEAR  HANDLE 

RP  GEAR 

Overspeed 

13 

OVERSPEED 

w 

THROTTLE  BACK 

RPOVRSPD 

Cabin  altitude 

14 

CABIN  ALT 

w 

COLUMN  FORWARD 

RP  CABN  ALT 

Left  generator  drive  oil 

16 

GEN  DRIVE  OIL 

c 

RP  ELEC 

DISCONNECT 

GENERATOR 

Gear  disagree 

17 

GEAR  DISAGREE 

c 

RP  GEAR 

RP  GEAR 

Right  system 
hydraulic  pressure 

18 

R  SYS  HYD  PRSR 

c 

RP  HYD 

CYCLE  RIGHT 

HYDRAULIC 

SYSTEM 

Antiskid  inoperative 

19 

ANTI  SKID  INOP 

c 

RP  ANTI  SKID 

RP  ANTI  SKID 

Left  air-conditioning 
pack  trip  off 

20 

L  PACK  TRIP 

c 

RP  ECS 

RP  ECS 

Forward  main  door  open 

21 

FWD  MAIN  DOOR 

c 

RP DOOR 

RPDOOR 

Right  engine 
oil  pressure  low 

22 

R  ENG  OIL PRSR 

c 

RP  ENG  STATUS 

RP  ENG  STATUS 

Anti-ice  inoperative 

23 

ANTI-ICE 

c 

RP  ANTI-ICE 

RP  ANTI-ICE 

Autospoiler  inoperative 

24 

AUTO-SPOILER 

c 

RP  AUTO-SPLR 

- 

Altitude  alert 

25 

ALTITUDE 

c 

RP  ALT 

RP  ALT  ALRT 

Left  bleed  off 

26 

L  BLEED  OFF 

A 

RP  ECS 

RP  ECS 

Galley  bus  off 

27 

GLY  BUS  OFF 

A 

RP  ELEC 

CYCLE  SWITCH 

Utility  bus  off 

28 

UTIL  BUS  OFF 

A 

RP  ELEC 

CYCLE  SWITCH 

Right  engine 
hydraulic  pump 

29 

R  ENG  HYD  PUMP 

A 

RP  HYD 

CYCLE  SWITCH 

Left  engine 
fire  detector 

30 

LENG  FIREDET 

A 

RP  FIRE 

RP  FIRE 

Left  brake  overheat 

31 

L BRAKE  OVHT 

A 

RPBRK 

RPBRK 

Right  forward 
fuel  pump 

32 

R  FWD  FUEL  PUMP 

A 

RP  FUEL 

CYCLE  SWITCH 

Forward  cabin  call 

33 

FWD  CABIN  CALL 

A 

RP  CABN  CALL 

RP  CABN  CALL 

SELCAL 

34 

SELCAL 

A 

RP  SELCAL 

RP SELCAL 

*  RP  =  response  panel. 


**  W  =  warning  C  =  caution  A  =  advisory 
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Table  5.4.4-1 .  Operational  and  System  Condition  for  Alerts  and  Their  Associated  Responses  (Concluded) 


Alert 

Alert 

code 

CRT  message 

Urgency 

level 

Pilot's  response 

Flight 

engineer's 

response 

Upper  yaw  damper 
failure 

35 

UPPER  YD  FAIL 

m 

RP  FLTCONTRL 

RP  FLTCONTRL 

Leading  edge  flaps 

36 

LE  FLAPS 

RP  FLTCONTRL 

RP  FLT CONTRL 

Air-conditioning 

pressure 

37 

AIRCOND/PRSR 

u 

RP  ECS 

RP  ECS 

Left  generator  off 

38 

L  GEN  OFF 

RP  ELEC 

CYCLE  SWITCH 

Left  bus  tie 

39 

L  BUS  TIE 

RP  ELEC 

CYCLE  SWITCH 

Right  electric 
hydraulic  pump 

41 

R  ELEC  HYD  PUMP 

RP  HYD 

CYCLE  SWITCH 

Autothrottle  disconnect 

43 

A/T  DISC 

RP  A/T 

RP  A/T 

Each  test  flight  was  31  minutes  in  length  and  contained  nine  alerts,  three 
from  each  urgency  level  (warning,  caution,  advisory).  The  alerts  were 
presented  on  a  schedule  of  three  minute  intervals;  however,  to  prevent  the 
pilot's  anticipation  of  the  alerts,  a  150  second  interval  around  each  three 
minute  mark  was  allocated  for  the  alerts  (mark  +  75  seconds  random).  The 
alerts  could  therefore  be  presented  as  close  together  as  30  seconds.  The 
times  were  chosen  at  random,  and  10  different  time  scenarios  were  developed. 
The  only  restriction  on  the  time  selection  was  that  no  alert  could  occur  after 
30  minutes  and  30  seconds  into  the  flight  to  permit  the  pilot  at  least  30 
seconds  to  respond  to  the  last  alert.  To  reduce  the  possibility  of 
influencing  the  data  by  the  order  in  which  the  alerts  were  presented,  10 
random  alert  orderings  were  developed  and  combined  at  random  with  the  time 
scenarios  to  produce  the  test  scenarios. 

Whenever  task  performance  is  measured  under  several  different  treatment 
conditions  over  an  extended  period  of  time,  learning  or  fatigue  may  affect 
performance  on  later  trials.  Care  was  taken  to  design  an  appropriate 
counterbalancing  scheme  to  prevent  carry-over  effects  from  differentially 
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affecting  the  performance  measures  for  the  different  treatment  conditions.  It 
should  be  noted,  therefore,  that  the  order  in  which  the  pilot  received  the 
experimental  treatments  was  also  randomly  assigned  to  prevent  order  bias  from 
confounding  the  results  (see  Table  5. 5. 1-1).  Immediately  prior  to  each  flight 
the  pilot  was  briefed  on  the  alerting  system  configuration  that  he  would  be 
using. 

The  daily  test  schedule  for  both  the  system  validation  tests  and  the 
time-critical  tests  is  presented  in  Table  5. 5.1-2;  all  pilots  were  to  fly  both 
tests.  Two  pilots  were  tested  each  week  spending  two  and  a  half  days  per 
pilot  in  the  simulation.  The  pilots  overlapped  on  the  afternoon  of  day  three; 
this  was  the  time  when  the  flight  engineer  system  validation  tests  were  run. 

A  total  of  12  pilots  completed  the  full  test  and  2  pilots  completed  just  the 
system  validation. 

The  test  participation  began  with  an  introduction  to  the  Visual  Flight 
Simulation  Facility  and  a  review  of  the  program.  The  pilots  were  briefed  on 
the  flight  plan  and  given  the  nominal  flight  path  parameters  (see  Figure 
5. 5.1-1).  They  were  encouraged  to  take  notes  on  their  briefing  sheet  and  to 
use  them  during  flight.  Following  the  briefing,  the  pilots  entered  the  cab 
for  instruction  on  the  operational  characteristics  of  the  simulator  and  the 
test  flight  tasks  (see  Appendix  F  for  the  briefing  checklist). 

The  pilots  were  informed  of  the  three  basic  tasks  to  be  carried  out  during 
each  flight.  The  first  involved  flying  the  simulator  from  take-off  to  landing 
on  the  specified  flight  plan.  The  second  was  responding  to  ATC  requests  for 
traffic  location.  Finally,  responses  to  the  alerts  were  to  be  made  by 
performing  the  prescribed  actions  associated  with  each  alert. 

Before  participating  in  the  data  collection  flights,  each  pilot  made  a  series 
of  practice  flights.  The  purpose  of  these  flights  was  twofold  -  to  acquaint 
the  pilots  with  the  flight  characteristics  and  dynamics  of  the  simulation 
airplane  model  and  the  flight  plan;  and  to  become  proficient  at  performing  the 
correct  alert  responses.  The  first  practice  flight  was  31  minutes  in  which 
the  complete  flight  pattern  was  flown.  There  were  no  alerts  to  distract  the 
pilots  during  this  flight.  The  instructions  on  how  to  respond  to  ATC 
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Table  5.5. 1-2.  Daily  Test  Schedule 


Odd-numbered  pilots 


Even-numbered  pilots 


Monday 

000-1:00 

0:30-2:30 

2:30-3:45 

3:45-4:30 

4:30-5:45 

5:45-6:00 

6:00-8:00 


Cab  warmup  and  preflight 
Pilot  training 

Plights  1  and  2 
Lunch 

Plights  3  and  4 
Break 

Plights  5  through  7 


Tuesday 
000-1 :00 
1:00-2:30 
2:30-2:45 
2:45-4:00 
4:00-4:45 
4:45-6:00 
6:00-6:15 
6:15-7:45 


Cab  warmup  and  preflight 

Plights  8  and  9 
Break 

Plights  10  and  11 
Lunch 

Plight  12  and  system  training 
Break 

Plights  13  and  14 


Wednesday 

000-1:00 

0:30-2:30 

2:30-3:45 

3:45-4:30 

4:30-5:30 

5:30-6:45 

6:45-7:00 

7:00-8:15 

8:15-9:30 


Plight  engineer  training 
Plights  15  and  16 
Break 

Plights  17  and  18 
Debrief 


Thursday 

000-1:00 

1:00-2:30 

2:30-2:45 

2:45-4:00 

4:00-4:45 

4:45-5:15 

5:15-5:30 

5:30-7:30 

Friday 

000-1:00 
1:00-2:30 
2:30-2:45 
2:45-3:15 
3:15— 4;30 


Cab  warmup  and  preflight 
Pilot  training 

Flights  13  and  14 
Lunch 

Plight  engineer  training 
Flights  17  and  18 
Break 

Flights  15  and  16 


Cab  warmup  and  preflight 
Plights  1  and  2 
Break 

Flights  3  and  4 
Lunch 

Plights  5  and  6 
Break 

Flights  7  through  9 


Cab  warmup  and  preflight 

Flights  10  and  1 1 
Break 
Flight  12 
Debrief 


Figure  5.5. 1-1.  Phase  3  Nominal  Flightpath 


requests  were  explained  during  the  practice  flight,  and  any  questions  the 
pilot  had  were  answered.  Since  the  pilots  had  the  most  problem  with  EADI 
symbology  (especially  the  gll deslope  box),  the  second  practice  flight  Included 
just  the  approach  from  10  miles  out.  After  these  two  familiarization  flights, 
the  alerts  were  demonstrated  “on  the  ground"  so  that  the  pilot  could 
concentrate  on  the  responses.  Then  the  alerts  were  repeated  and  the  pilots 
were  asked  to  respond  to  them  by  performing  the  corrective  action.  The  pilots 
then  flew  a  full  flight  with  all  the  system  components  working.  The  time  for 
training  was  two  and  one  half  hours. 

The  test  day  consisted  of  eight  flights  with  approximately  4  hours  of  flying. 
Brief  rest  periods  were  taken  throughout  the  day  in  an  effort  to  reduce 
fatigue. 

Upon  completing  the  data  collection  flights  the  pilots  participated  in  a 
debriefing  session.  First,  a  debriefing  questionnaire  was  completed  (see 
Appendix  G).  Their  Impressions  of  the  advanced  concepts  and  the  application 
of  the  concepts  were  solicited.  The  formal  debriefings  Included  an  informal 
discussion  between  the  pilots  and  experimenter  and  relevant  pilot  comments 
were  recorded  for  further  evaluation. 

The  validation  tests  of  the  flight  engineer  alerting  system  concept  were 
conducted  on  Wednesday  afternoon  when  both  pilots  were  present.  The  pilots 
were  given  a  training  session  at  the  flight  engineer  station  to  familiarize 
them  with  the  expected  response  actions  to  the  alerts  and  the  procedures  to  be 
followed  during  the  test  flights.  After  the  training  session  the  pilots 
alternated  positions  for  the  afternoon  flights. 

5.5.2  TIME-CRITICAL  WARNING  TESTS 

The  variables  Investigated  In  the  time-critical  tests  are  described  In  Section 
5.2.2.  As  In  the  validation  tests,  all  variables  not  tested  were  held 
constant  or  controlled  to  prevent  biasing  or  confounding  the  results.  The 
values  for  the  variables  are  presented  In  the  previous  section. 


As  In  the  validation  test,  each  test  flight  was  31  minutes  In  length  and 
contained  nine  alerts.  Four  of  the  alerts  were  defined  as  time-critical 
warnings,  four  were  warnings  which  were  not  "time-critical" ,  and  one  alert  was 
a  multiple  problem  In  which  at  least  one  alert  from  each  urgency  level 
appeared  at  the  same  time.  The  time-spacing  and  selection  of  alerts  were 
performed  In  the  the  same  manner  as  for  the  validation  tests.  Eleven  alert 
scenarios  were  developed  to  reduce  the  possibility  of  incuring  order  effects. 
The  presentation  schedule  for  the  treatment  conditions  Is  presented  In  Table 
5. 5. 1-1. 

Other  than  those  noted  above,  there  were  no  other  procedural  differences 
between  the  time-critical  tests  and  the  system  validation  tests. 

5.6  PERFORMANCE  MEASURES 

The  performance  measures  used  In  the  tests  fell  Into  two  categories  -  those 
associated  with  the  flight  task  and  those  associated  with  the  alert  response 
tasks.  The  parameters  that  reflect  how  well  the  pilot  was  performing  the 
flight  task  were  crosstrack  deviation  from  the  flight  path,  altitude 
deviations,  wheel  and  column  reversals,  landing  performance,  and  speed  and 
accuracy  of  detection  of  the  outside  visual  targets.  The  parameters  were 
especially  Important  for  the  time  period  Immediately  around  the  alerts  because 
they  provide  a  measure  of  the  efficiency  and  effectiveness  of  the  pilot  in 
performing  the  flight  task.  A  second  set  of  dependent  variables,  used  to 
quantify  the  responses  to  the  alerting  system.  Included  the  time  and  accuracy 
of  alert  detection,  the  time  and  accuracy  of  the  response  to  the  alert,  the 
system  component  used,  and  the  sequence  In  which  the  pilot  performed  the  alert 
cancellation. 

Finally,  subjective  data  expressing  the  pilot's  opinions  about  the  various 
alerting  system  characteristics  were  gathered  for  all  test  configurations. 

The  pilots  were  asked  to  comment  on  and  rate  the  effectiveness  of  the 
candidate  systems,  clarity  of  the  messages,  format  and  system  components. 


5.7  DATA  REDUCTION  AND  ANALYSES 


The  data  obtained  In  the  Phase  3  testing  falls  Into  two  general  categories  > 
objective  (or  performance)  data  and  subjective  (questionnaire/debriefing) 
data.  A  time-based  tabulation  of  all  events  that  occurred  in  the  cab,  switch 
and  light  states,  displayed  messages  and  fault  situation  Initiation,  was 
generated  from  the  data.  From  this  tabulation,  sums,  means  and  standard 
deviations  were  calculated  for  all  performance  variables.  The  performance  was 
analyzed  with  respect  to  all  the  alerts  and  was  also  partitioned  Into  the 
various  alert  categories.  Analyses  of  variance  were  performed  on  the  reduced 
data  to  determine  If  the  various  treatment  conditions  had  a  differential 
affect  upon  performance. 

The  statistical  model  for  the  system  validation  test  was  a  straight  comparison 
between  the  conventional  alerting  system  and  the  advanced  alerting  concepts. 
This  resulted  In  a  two  factor  analysis  of  variance  with  a  single  testable 
effect  -  the  alerting  system. 

The  time-critical  tests  on  the  other  hand,  had  a  much  more  complex  model  since 
they  were  of  a  mixed  design.  All  pilots  had  the  treatment  conditions 
associated  with  three  of  the  variables,  but  the  fourth  variable  (message 
format)  divided  the  pilots  Into  two  groups.  The  model  and  source  table  for 
these  analyses  are  presented  In  Table  5.7-1. 

Since  validation  testing  requires  that  system  developers  be  very  sure  before 
they  reject  any  candidate  system  concept,  and  since  the  time  critical  tests 
were  exploratory  In  nature,  an  error  probability  of  .10  was  selected  as  a  test 
for  significance  for  the  statistical  tests  performed  on  both  experiments. 

5.7.1  SYSTEM  VALIDATION  TESTS— EXPERIMENTAL  HYPOTHESIS 

The  following  were  the  hypotheses  upon  which  the  validation  tests  were  based. 
Each  of  the  hypotheses  can  be  equally  stated  for  all  levels  of  alerting 
urgency,  l.e.,  warning,  caution  and  advisory. 


Table  5.7- 1.  Sample  of  Analysis  of  Various  Model  and  Summary  Table  for  a  Factorial 
Experiment  With  Repeated  Measures  on  Some  of  the  Factors 
Note;  The  example  is  a  two-factor  experiment  with  repeated  measures  on  one  factor. 


Modal 

*ijk  = 

/<  +  +  Tk(i)  +  V  "V  ^i«i)  +  eMii» 

Summary  Tabla  j 

Source 

Expected  mean  square 

F  ratio 

A 

ce2  +  ba  os2  +  nb  o^2 

MSA/MSSub 

Subject  within  A 

oe2  +  ba  cs2 

B 

9  2  2 

°e  +  8  ffBs  +nafllB 

MSg/MSgs 

A  x  B 

°e2  +  a0Bs2  +  ,'0AB2 

M**AB^®Bs 

B  x  subjects  within  A 

2  2 

V+aoBs 

1.  There  is  no  difference  In  the  pilots  detection  time  between  the 
conventional  and  advanced  alerting  systems  A  or  B. 

2.  Pilot  response  time  is  not  affected  by  the  alerting  system  used. 

3.  There  is  no  difference  in  the  number  of  missed  alerts  between  the 
conventional  alerting  system  and  concepts  A  or  B. 

4.  The  type  of  alerting  system  used  has  no  effect  on  the  pilot's  flight 
performance. 

5.  Flight  performance  is  not  affected  by  the  urgency  of  the  alert. 

6.  There  is  no  difference  in  the  flight  engineer's  response  time  between  the 
conventional  and  advanced  alerting  systems. 
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7.  There  Is  no  difference  In  the  number  of  missed  alerts  by  the  flight 
engineer  using  conventional  and  advanced  alerting  systems. 

8.  The  pilots  have  no  preference  for  either  the  conventional  or  advanced 
visual  displays. 

9.  The  pilots  have  no  preference  between  the  conventional  discrete  sounds 
and  the  advanced  master  aural  alerts. 

10.  The  pilots  have  no  preference  between  verbal  and  non-verbal  alerts. 

5.7.2  TIME-CRITICAL  WARNING  TESTS— EXPERIMENTAL  HYPOTHESES 

The  following  are  the  hypotheses  upon  which  the  time-critical  warning  tests 

were  based: 

1.  There  Is  no  difference  In  detection  times  between  time-critical  and  other 
warni ngs . 

2.  There  Is  no  difference  In  response  times  between  time-critical  and  other 
warnings. 

3.  The  error  distribution  of  the  time-critical  warnings  does  not  change  for 
different  presentation  formats,  message  content,  voice  formats,  or 
display  locations. 

4.  Detection  or  response  times  for  time-critical  warnings  are  the  same  when 
the  location  of  the  visual  information  Is  at  a  30°  visual  angle  or  at  a 
15°  visual  angle. 

5.  There  Is  no  difference  In  the  detection  or  response  times  for  the 
time-critical  warnings  among  the  three  presentation  formats  - 
alphanumeric,  graphic  or  both. 

6.  There  Is  no  difference  In  the  detection  or  response  times  for 
time-critical  warnings  between  status  and  guidance  messages. 


7.  The  presence  or  absence  of  a  voice  alert  has  no  effect  on  the  detection 
or  response  times  for  time-critical  warnings. 

8.  Flight  performance  Is  not  affected  by  alert  type,  time-critical  warnings 
or  non-time-critical  warnings. 

9.  Flight  performance  Is  not  affected  by  the  location  of  the  time-critical 
display. 

10.  Flight  performance  is  not  affected  by  the  presentation  formats  of  the 
time-critical  warnings. 

11.  Flight  performance  is  not  affected  by  the  message  formats  of  the 
time-critical  warnings. 

12.  Flight  performance  is  not  affected  by  the  presence  or  absence  of  a 
time-critical  voice  alert. 

13.  The  pilots  had  no  preference  for  the  time-critical  warning  display 
location,  presentation  format,  message  format  or  voice. 
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5.8  TEST  RESULTS 


Although  some  of  the  results  reported  in  the  following  sections  as  being 
statistically  significant  may  appear  to  be  of  insufficient  magnitude  to  be  of 
practical  importance,  this  may  be  a  false  assessment  of  the  results,  however, 
due  to  the  nature  of  the  tests.  It  must  be  kept  in  mind  that  the  pilots  knew 
that  alerts  were  going  to  occur  during  the  flight,  which  would  result  in  a 
response  that  was  faster  than  would  normally  occur.  Since  the  speed  of  a 
response  is  bounded  on  the  low  side  by  physical  parameters,  (i.e.,  recognition 
and  reaction  time)  smaller  differences  between  treatments  are  to  be  expected 
when  working  with  shorter  response  times.  It  is  also  expected  that  any 
differences  discovered  in  this  type  of  test  would  be  magnified  under  actual 
flight  conditions. 

5.8.1  SYSTEM  VALIDATION  RESULTS 

5.8.1. 1  DETECTION  TIMES 

Alert  detection  time  was  defined  as  the  time  between  the  onset  of  the  alert 
and  the  depression  of  the  left  hand  thumb  switch  by  the  pilot.  The  analysis 
of  variance  summary  tables  are  presented  in  Table  5. 8. 1.1-1. 

The  advanced  systems  consistently  produced  shorter  mean  detection  times  than 
the  conventional  system.  These  differences  in  times  with  the  system  A  were 
significant  for  both  Warnings  (F  =  5.25  df  1,9)  and  advisories  (F  =  3.4  df 
1,9).  Although  the  time  differences  were  small  (1.88  seconds  versus  1.48 
seconds  and  2.15  seconds  versus  1.81  seconds  respectively)  the  pilots 
consistency  led  to  the  significance  of  the  data.  The  pilots  who  were  tested 
using  system  B,  however,  were  more  variable  in  their  detection  with  a  standard 
deviation  of  1.57  seconds  as  compared  to  0.26  second  for  system  A.  The  result 
suggests  that  the  use  of  the  voice  selection  option  added  variability  to  the 
data  which  in  turn  made  it  more  difficult  to  identify  true  differences.  System 
B  pilots  also  took  longer  on  the  average  to  detect  the  alerts  both  for  the 
advanced  and  conventional  systems  (2.02  seconds  and  2.53  seconds  versus  1.54 
and  1.86  seconds).  The  result  of  the  greater  variability  in  responses  was 
that  even  though  the  differences  between  advanced  and  conventional  system 
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response  times  for  warning  and  advisories  was  larger  with  System  B  than  with 
System  A  the  differences  were  not  significant.  Illustrations  of  the  detection 
times  for  Systems  A  and  B  are  presented  in  Figures  5.8. 1.1-1  and  5.8. 1.1-2. 


The  pilots  varied  widely  on  their  use  of  voice  in  System  B.  Both  extremes 
were  observed.  One  pilot  never  used  the  voice  and  one  pilot  initiated  it  for 
all  the  warnings  and  cautions.  For  those  pilots  who  used  the  voice,  warnings 
were  always  requested  equal  to  or  more  often  than  caution  (25  to  17). 

5.8. 1.2  VALIDATION  RESPONSE  TIMES 


Pilot  response  time  was  defined  as  the  time  from  the  alert  onset  to  the 
completion  of  the  response  perscribed  for  that  alert.  The  analysis  of 
variance  summary  table  for  the  system  validation  response  times  is  presented 
in  Table  5.8. 1.2-1. 

As  with  detection  times,  the  advanced  alerting  systems  consistently  produced 
shorter  response  times  than  did  the  conventional  system.  Both  advanced 
systems  produced  significantly  shorter  (F  *  3.7  &  3.5  df  1,14  and  1,9)  mean 
response  times  to  cautions  (5.0  and  5.4  seconds)  than  did  the  conventional 
(6.6  and  7.0  seconds) . 

There  was  no  difference  in  the  response  times  to  warnings  or  advisories 
between  the  advanced  and  conventional  systems.  The  response  times  for  System 
A  and  B  are  graphically  compared  with  the  conventional  system  in  Figures 
5.8. 1.2-1  and  5.8. 1.2-2,  respectively. 

Response  times  were  also  obtained  as  a  validation  of  the  use  of  an  advanced 
display  for  the  Flight  Engineer.  The  mean  response  time  to  warnings  was 
significantly  shorter  (F  =  5.19  df  1,9)  for  the  advanced  display  (5.52 
seconds)  than  for  the  conventional  alerting  scheme  (8.36  seconds).  Although 
not  statistically  significant,  the  cautions  were  responded  to  faster  with  the 
advanced  display  (7.8  seconds  versus  9.1  seconds).  The  relationship  is 
illustrated  in  Figure  5.8. 1.2-3. 
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Table  5.8.1. 1-1.  System  Validaton  Detection  Time  Summary  (Composite  of  the  ANOVA's ) 


Source 

Sum  of 
squares 

Degrees  of 
freedom 

Mean 

square 

F 

Probability  F 
exceeded 

System  A 

Warning 

0.794 

1 

0.794 

5.25 

0.04  * 

Error 

1.359 

9 

0.151 

Caution 

0.124 

1 

0.124 

2.01 

0.189 

Error 

0.545 

9 

0.061 

Advisory 

0.560 

1 

0.560 

3.40 

0.10* 

Error 

1.482 

9 

0.165 

System  B 

Warning 

1.244 

1 

1.244 

1.08 

0.339 

Error 

6.917 

6 

1.152 

Caution 

0.110 

1 

0.110 

0.05 

0.817 

Error 

11.315 

6 

1.885 

Advisory 

2.081 

1 

2.081 

0.39 

0.552 

Error 

31.506 

6 

5.251 

Table  5.8.1. 2-1.  System  Validation  Response  Time  Summary  (Composite  of  the  ANOVA's) 


Source 

Sum  of 

squares 

Degrees  of 
freedom 

Mean 

square 

F 

Probability  F 
exceeded 

System  A 

Warning 

2.787 

1 

2.787 

1.54 

0.23 

Error 

25.351 

14 

1.811 

Caution 

19.214 

1 

19.214 

3.66 

0.07* 

Error 

73.413 

14 

5.243 

Advisory 

0.037 

1 

0.C  / 

0.01 

0.92 

Error 

49.877 

14 

3.562 

System  B 

Warning 

0.820 

1 

0.820 

1.22 

0.29 

Error 

6.059 

9 

0.673 

Caution 

12.319 

1 

12.319 

3.52 

0.09* 

Error 

31.497 

9 

3.50 

Advisory 

0.024 

1 

0.024 

0.008 

0.93 

Error 

25.892 

9 

2.876 

‘Significant  at  the  .10  level  or  better. 
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DETECTION  TIME  (sec) 


'Significant  at  0.10  level. 

Figure  5.8. 1. 1- 1.  Candidate  System  A— Detection  Time 
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ALERT  CATEGORY 
‘Significant  at  0.10  level. 

Figure  5.8. 1. 1-2.  Candidate  System  B— Detection  Time 


M 


✓ 


* 

_ I _ I _ I _ 

WARNING  CAUTION*  ADVISORY 

ALERT  CATEGORY 

’Significant  at  O.tO  level. 


Figure  5.8. 1.2-1.  Candidate  System  A— Response  Time 


ADVISORY 


WARNING  CAUTION* 

ALERT  CATEGORY 
'Significant  at  0.10  level. 

Figure  5.8. 1.2-2.  Candidate  System  B-Response  Time 


5.8. 1.3  MISSED  ALERTS 


Each  alert  remained  active  for  30  seconds.  If,  after  that  time,  the  pilot  had 
not  made  the  specified  response  the  alert  was  considered  missed.  Out  of  216 
alerts  presented  with  the  conventional  system  7  were  missed  or  3  percent.  Of 
those  7  missed  alerts  3  were  warnings  and  3  cautions.  The  warning  alerts 
missed  all  were  aural -presentation-only  type  alerts  (e.g.,  overspeed  or  cabin 
altitude). 

The  advanced  systems  had  only  two  missed  alerts  between  them  (one  advisory 
with  System  A  and  one  caution  with  System  B)  or  less  than  one  percent. 

The  flight  engineers  exhibited  a  larger  missed  alert  rate.  Using  their 
conventional  system  they  missed  30  alerts  out  of  90  possible  or  thirty-three 
percent.  The  breakdown  of  the  missed  alerts  was  10  warnings,  11  cautions  and 
9  advisories.  Using  the  advanced  flight  engineer's  system  however,  reduced 
the  number  of  missed  alerts  to  7  or  about  8  percent.  The  were  also  evenly 
dlstibruted,  with  2  warnings,  3  cautions,  and  2  advisories.  These  data  are 
presented  graphically  in  Figure  5. 8. 1.3-1. 

5.8. 1.4  PILOT  PREFERENCES 

The  debriefing  questionnaire  is  presented  in  Appendix  G.  All  of  the  pilots 
preferred  the  advanced  systems  over  the  conventional.  None  of  the  pilots  felt 
that  the  conventional  system  was  more  attention-getting  or  provided  easier 
problem  identification.  Nlnty-three  percent  of  the  pilots  found  the  advanced 
visual  system  easier  to  use  and  eighty-five  percent  responded  in  the  same 
manner  for  the  master  aural  sounds.  Seventy  percent  of  the  pilots  Indicated  a 
preference  for  a  voice  system  to  augment  the  visual  alerts  while  eight  percent 
preferred  the  master  aural  alert  only.  Eight  percent  of  the  pilots  also  felt 
that  the  master  aural  sound  was  a  more  effective  attention-getter  than  the 
combination  of  a  sound  and  a  voice.  None  of  the  pilots  felt  that  the 
Intensity  level  of  the  alerts,  both  master  sound  and  voice,  (8dB  above 
ambient)  was  not  loud  enough.  Seventy-seven  percent  Indicated  that  8dB  was 
the  appropriate  level,  however,  twenty-three  percent  said  it  was  too  loud. 


WARNING  CAUTION  ADVISORY 

ALERT  TYPE 

Figure  5.8. 1.3 - 1.  Flight  Engineer  System  Validation— Missed  Alerts 


The  pilots  were  then  asked  to  rate  system  features  on  the  following  5  point 
scale: 

1  *  Unacceptable  -  major  changes  necessary 

2  *  Poor  -  major  changes  recommended 

3  =  Fair  -  minor  changes  recommended 

4  =  Good  -  minor  changes  beneficial 

5  =  Excellent  -  no  changes  recommended 

All  of  the  characteristics  of  the  visual  information  display  had  an  average 
rating  of  4.0  or  better.  These  characteristics  included,  location,  character 
size,  alert  color,  brightness,  character  separation,  message  content  and  the 
new  message  indicator.  The  auditory  display  also  received  high  ratings.  The 
number  of  master  aural  alerts  received  an  average  rating  of  4.4  with  both  the 
warning  and  advisory  sounds  receiving  ratings  above  4.0  (4.2  A  4.1 
respectively).  The  caution  sound  had  a  mean  rating  of  3.1  with  the  majority 
of  the  pilots'  comments  centered  around  the  urgency  of  the  sound  and  its 
perceived  resemblance  of  the  warning  sound.  The  voice  alerts  had  a  mean 
rating  of  4.5  or  better  on  ease  of  use,  lack  of  confusion  with  other 
communication,  message  content  and  voice  type  (female).  The  following 
characteristics  of  voice  received  a  mean  rating  of  4.0  to  4.5; 
intelligibility,  loudness  and  repetition  rate. 

Finally,  the  pilots  were  asked  to  list  the  5  things  that  they  most  liked  about 
the  advanced  alerting  systems  and  the  5  things  they  liked  least.  Eighty-five 
percent  of  the  pilots  listed  the  central  location  for  all  alerts  as  a  highly 
preferred  feature.  Other  features  that  were  listed  as  "most  liked"  items 
were:  unique  tone  for  warning,  caution  and  advisory  (70%);  voice  on  cautions 
(54%);  volume  of  the  auditory  components  (31%);  separate  colors  for  each  alert 
category  (23%);  features  receiving  less  than  twenty  percent  of  the  pilots 
responding  are  not  included  here. 

Only  one  feature  of  the  advanced  systems  was  liked  least  by  more  than  twenty 
percent  of  the  pilots,  and  that  was  the  seeming  urgency  of  the  caution  sound 
(39%). 
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5.8.2  TIME-CRITICAL  TEST  RESULTS 


5.8.2. 1  DETECTION  TIMES 

The  analysis  of  variance  summary  table  for  the  time-critical  alert  detection 
time  is  presented  in  Table  5. 8. 2. 1-1.  None  of  the  main  effect  variables 
(location,  presentation  format,  voice,  or  message  information)  had  a 
measurable  effect  on  the  mean  detection  time.  However,  there  was  a 
significant  interaction  (F  *  2.87  df  2,16)  between  the  presentation  format  and 
the  message  content.  This  interaction  indicates  that  the  mean  detection  times 
are  significantly  shorter  for  the  guidance  information  if  graphics  are  used 
(1.64  seconds  and  1.08  seconds)  than  for  the  status  messages  (1.96  seconds  and 
1.98  seconds).  This  effect  is  illustrated  in  Figure  5.8.2. 1-1. 

There  was  no  measurable  difference  between  the  detection  of  time-critical  and 
non-time-critical  warnings.  In  Figure  5. 8. 2. 1-1  the  comparison  is  shown 
between  the  non-time-critical  warnings  (A)  which  were  alphanumeric  status 
messages  and  the  corresponding  time-critical  warning  (1.57  seconds  versus  1.52 
seconds).  Finally,  the  presentation  media  for  the  time-critical  warnings  had 
no  effect  on  the  detection  of  the  non-time-critical  warnings.^ 

5.8.2. 2  RESPONSE  TIMES 

The  analysis  of  variance  summary  table  for  the  response  times  to  time-critical 
warnings  is  presented  In  Table  5. 8. 2. 2-1.  The  main  effect  attributed  to 
message  information  type  was  significant  (F  =  3.59  df  1,10)  with  the  mean 
response  time  for  guidance  messages  (2.58  seconds)  being  shorter  than  for 
status  messages  (2.95  seconds).  No  other  main  treatment  conditions  had  by 
themselves  a  measurable  effect  on  response  time.  There  were  interactive 
effects,  however,  which  did  provide  measurable  differences.  The  Interaction 
between  presentation  format  (alphanumeric,  graphic  and  both)  and  the  type  of 
Information  presented  (status  or  guidance)  was  very  high  (F  *  4.49  df  2,20). 
This  Interaction  Is  presented  In  Figure  5. 8. 2.2-1.  It  can  be  seen  that  the 
guidance  messages  presented  graphically  are  responded  to  faster  (2.51  seconds) 
than  any  presentation  of  status  Information.  Graphic  presentation  of  status 
information  resulted  in  the  longest  mean  response  time  (3.04  seconds). 
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Table  5.8.2. 1-1.  Summary  Table  for  Time-Critical  Detection  Time 


Source 

Sum  of 

Degrees  of 

Mean 

F 

Probability  F 

squares 

freedom 

square 

exceeded 

Mean 

296.43463 

1 

296.43463 

50.26492 

0.000 

Message  content 

5.12679 

1 

5.12679 

0.869 

0.37 

Error 

47.17956 

8 

5.89745 

Display  location 

0.28045 

1 

0.28045 

0.363 

0.56 

Display  location  x  message  content 

0.29145 

1 

0.29145 

0.377 

0.55 

Error 

6.17637 

8 

0.77205 

Presentation  format 

2.52754 

2 

1.26377 

2.203 

0.14 

Presentation  format  x  messarr  <-ontent 

3.28749 

2 

1.64374 

2.865 

0.08* 

Error 

9.17772 

16 

0.57361 

Display  location  x  presentation  format 

2.12321 

2 

1.06160 

0.804 

0.46 

Display  location  x  presentation  format 

2.11918 

2 

1.05959 

0.803 

0.46 

x  message  contei  t 

Error 

21.10847 

16 

1.31928 

Voice 

0.50955 

1 

0.50955 

1.028 

0.34 

Voice  x  message  content 

0.16812 

1 

0.16812 

0.339 

0.57 

Error 

3.96502 

8 

0.49563 

Display  location  x  voice 

1.06507 

1 

1.06507 

1.542 

0.24 

Display  location  x  voice  x  message  content 

0.06923 

1 

0.06923 

0.100 

0.76 

Error 

5.52552 

8 

0.69069 

Presentation  format  x  voice 

0.54523 

2 

0.27261 

0.446 

0.64 

Presentation  format  x  voice 

0.83738 

2 

0.41869 

0.685 

0.51 

x  message  content 

Error 

9.77446 

16 

0.61090 

Display  location  x  presentation  format 

0.30163 

2 

0.15081 

0.601 

0.56 

x  voice 

Display  location  x  presentation  format 

0.33872 

2 

0.16936 

0.675 

0.52 

x  voice  x  message  content 

Error 

4.01502 

16 

0.25094 

‘Significant  at  the  .10  level  or  better. 
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DETECTION  TIME  (sec) 


PRESENTATION  FORMAT 
▲  Average  detection  time  for  non-time-critical  warnings. 


Figure  5.8.2. 1-1.  Detection  Time  interaction  Between  Presentation 
Format  and  Message  Content 
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Table  5.8.2. 2-1.  Summary  Table  for  Time-Critical  Response  Time 


Source 

Sum  of 
squares 

Degrees  of 
freedom 

Mean 

square 

F 

Probability  F 
exceeded 

Mean 

1101.20557 

1 

1101.20557 

606.99744 

0.000 

Message  content 

1.30474 

1 

1.30474 

3.596 

0.09* 

Error 

3.62837 

10 

0.36284 

Display  location 

0.00388 

1 

0.00388 

0.005 

0.94 

Display  location  x  message  content 

0.44745 

1 

0.44745 

0.633 

0.44 

Error 

7.06132 

10 

0.70613 

Presentation  format 

0.26088 

2 

0.13044 

0.448 

0.64 

Presentation  format  x  message  content 

2.61547 

2 

1.30773 

4.497 

0.02* 

Error 

5.81526 

20 

0.29076 

Display  location 

0.31824 

2 

0.15912 

0.735 

0.49 

Display  location  x  presentation  format 

1.20346 

2 

0.60173 

2.731 

0.07* 

x  message  content 

Error 

4.32581 

20 

0.21629 

Voice 

0.12244 

1 

0.12244 

0.412 

053 

Voice  x  message  content 

0.00375 

1 

0.00375 

0.012 

0.91 

Error 

2.96806 

10 

0.29681 

Display  location  x  voice 

0.00839 

1 

0.00839 

0.023 

0.88 

Display  location  x  voice  x  message  content 

0.86289 

1 

0.86289 

2.429 

0.18 

Error 

3.55166 

10 

0.35517 

Presentation  format  x  voice 

0.04943 

2 

0.02472 

0.110 

0.89 

Presentation  format  x  voice 

0.19669 

2 

0.09835 

0.439 

0.65 

x  message  content 

Error 

4.47214 

20 

0.22361 

Display  location  x  presentation  format 

1.03525 

2 

0.51763 

2.238 

0.13 

x  voice 

Display  location  x  presentation  format 

1.05758 

2 

0.52879 

2.868 

0.08* 

x  voice  x  message  content 

Error 

3.64682 

20 

0.18234 

'Significant  at  the  .10  leva)  or  batter. 
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’Significant  at  0.10  level. 


Figure  5.8.2.2-1.  Response  Time  interaction  Between  Presentation 
Format  and  Message  Content 
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3.2 


FORMAT 

DISPLAY  LOCATION-*  SECONDARY  PRIMARY 

▲  Average  response  to  non-time-critical  alerts. 

*  Significant  at  0.10  level 

Figure  5.8.2.2-2.  Mean  Response  Time  as  a  Function  of  the  Interaction  Between  Display 
Location,  Presentation  Format,  and  Message  Content 


The  addition  of  the  display  location  to  presentation  format  and  message 
information  also  resulted  in  a  significant  interaction  (F  =  2.73  df  2,20). 
Figure  5.8. 2.2-2  illustrates  this  interaction.  It  also  illustrates  the 
significant  difference  between  the  mean  response  times  for  non-time-critical 
alerts  (3.15  seconds)  ana  the  corresponding  time-critical  alert  (2.71 
seconds).  When  guidance  information  was  presented  graphically  (or  graphically 
with  alphameric s)  in  the  pilot's  primary  field  of  view,  the  resulting  mean 
response  times  (2.41  seconds  and  2.61  seconds)  were  significantly  shorter  than 
using  status  information  (2.95  seconds).  Presenting  status  information 
graphically  in  the  pilot's  secondary  field  of  view  resulted  in  the  longest 
mean  response  time  (3.09  seconds). 
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Finally,  the  partitioning  of  voice  out  of  this  interaction  also  produced 
significant  effects  {F  =  2.87  df  2,20).  This  four-way  interaction  is 
illustrated  in  Figure  5. 8. 2. 2-3.  With  these  data  the  treatment  condition 
which  produced  the  shortest  response  time  can  be  identified.  When  the  pilots 
were  presented  guidance  information  graphically  in  their  primary  field  of  view 
accompanied  by  voice  they  responded  the  fastest  (2.29  seconds).  Graphic 
presentation  of  the  status  information  resulted  in  the  slowest  mean  responses 
for  both  voice  (3.19  seconds)  and  no  voice  (3.22  seconds)  conditions.  As  can 
be  seen  in  the  illustration,  except  for  the  status  alerts  in  the  secondary 
field  of  view  with  voice,  the  alphanumeric  format  and  the  combined 
alpha-graphic  format  resulted  in  essentially  the  same  response  times  for  their 
particular  treatment  conditions. 

5.8.2.3  MISSED  ALERTS 

The  pilots  responded  to  all  of  the  alerts  both  time-critical  and 
non-time-critical  warnings  during  the  time-critical  testing.  This  result 
corresponds  with  the  data  from  the  system  validation  tests  in  which  no 
warnings  were  missed  with  the  advanced  systems. 

5. 8. 2.4  PILOT  PREFERENCES 

The  debriefing  questionnaire  is  presented  in  Appendix  G.  Eighty-three  percent 
of  the  pilots  said  that  for  time-critical  warnings  they  would  prefer  guidance 
information  to  facilitate  the  correct  response.  There  was  no  clear  preference 
for  the  alert  location.  Eighty-five  percent  of  the  pilots  said  that  they 
could  see  the  time-critical  alerts  equally  as  fast  in  the  primary  and 
secondary  fields-of-view.  Sixty-nine  percent  had  no  preference  for  either 
location.  However,  during  the  debriefing  interview  70%  of  the  pilots  said 
that  if  the  alert  message  was  providing  guidance  information  then 
consideration  should  be  given  to  presenting  it  on  the  primary  guidance 
instrument,  the  EAOI. 

Sixty-one  percent  of  the  pilots  felt  that  the  graphics  used  for  the  test  were 
too  cluttered.  None  of  the  pilots  rated  the  graphics  alone  as  easiest  to  use, 
promoting  fastest  response,  promoting  the  fewest  errors  or  as  the  preferred 
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presentation  format.  They  reported  that  the  graphic  presentation  for  the 
guidance  Information  did  not  put  enough  emphasis  on  the  guidance  element  and 
thus  were  less  effective  than  they  could  have  been.  Although  they  had 
difficulty  with  the  graphic  presentation,  46%  of  the  pilots  said  that  they 
would  like  to  see  a  combination  of  alphanumeric  and  graphic  formats. 
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6.0  DISCUSSION  AND  CONCLUSIONS 
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The  discussion  of  the  system  simulation  tests  will  focus  on  the  validation  of 
the  candidate  alerting  concepts  and  the  design  features  which  affect  the 
pilot's  response  to  time-critical  alerts. 

6.1  SYSTEM  VALIDATION 

In  the  system  validation  tests,  two  candidate  advanced  alerting  system 
concepts  were  compared  to  a  conventional  alerting  system.  In  all  areas  of 
measurement,  the  advanced  systems  resulted  in  pilot  performance  that  was  as 
good  or  better  than  the  conventional  system.  This  was  true  even  for  the 
response  times  with  System  B  where  the  pilots  were  using  the  optional  voice 
much  more  than  would  be  expected  after  the  novelty  wore  off  (50%  of  the  time). 
Therefore,  it  can  be  concluded  that  the  candidate  alerting  concepts  appear  to 
•represent  viable  alerting  system  designs. 

The  alerting  system  has  four  primary  functions,  to  attract  the  crews 
attention.  Identify  the  urgency  level  of  the  alert,  direct  crew  attention  to 
the  source  of  the  problem,  and  provide  feedback  on  the  adequacy  of  the 
corrective  action.  It  was  expected  that  the  changes  in  the  system  components 
between  the  advanced  and  conventional  systems  should  produce  detectable 
changes  in  the  ability  of  the  systems  to  perform  these  functions. 

If  detection  time  is  defined  as  a  measure  of  the  attention-getting  function  of 
the  system,  the  results  indicate  that  there  was  a  significant  improvement  of 
the  advanced  over  the  conventional  systems.  Warning  and  advisory  alerts  had 
significantly  shorter  mean  detection  time  with  System  A  than  with  the 
conventional  system.  Larger  differences  in  mean  detection  time  were  produced 
by  System  B;  however,  the  increase  in  pilot  variability  caused  these 
differences  to  be  classified  as  insignificant.  The  data  suggest  that  the 
higher  variability  was  the  result  of  using  the  voice  selection.  Therefore, 
the  results  of  the  System  B  validation  tend  to  support  those  from  System  A. 
When  considering  the  attention-getting  features  of  the  warning  level  alerts, 
the  advanced  and  conventional  systems  differed  in  both  the  visual  and  auditory 
components.  The  advanced  systems  had  a  single  master  light  and  sound  for  all 


warnings,  and  the  conventional  had  different  lights  and  sounds  for  each  alert. 
The  reduction  In  the  number  of  sounds  should  not  In  theory  aid  in  detection 
but  rather  reduce  confusion  In  identification.  Therefore,  the  consolidation 
of  all  the  discrete  warning  lights  into  a  single  master  alert  located  In  the 
pilot's  primary  field  of  view  seems  to  have  resulted  in  an  Improvement  of  the 
detection  performance.  This  hypothesis  is  supported  by  the  results  of  the 
caution  alerts.  Both  the  conventional  and  advanced  systems  had  a  single 
master  visual  light  located  in  the  pilot's  primary  field  of  view.  The 
advanced  systems  also  had  a  master  caution  sound  to  attract  the  pilot's 
attention.  The  lack  of  measurable  difference  in  the  mean  detection  times  for 
caution  alerts  between  the  systems  indicates  that  the  master  visual 
attention-getter  is  being  used  by  the  pilots.  In  the  absence  of  a  master 
light  in  the  primary  field  of  view  (or  the  existance  of  a  visual  workload 
reducing  the  probability  of  seeing  the  light)  the  master  aural  provides  the 
necessary  attention-getting  function.  This  is  demonstrated  by  the  mean 
detection  times  for  advisories.  Even  the  abbreviated  sound  (single  stroke 
chime)  used  for  the  advisories  in  the  advanced  system  was  sufficient  to  result 
in  significantly  shorter  mean  detection  times  than  were  found  with  the 
conventional  system  which  had  no  master  attention-getters  for  advisories. 

In  a  recent  study  Boucek,  et  al.,  (1980)  found  that  pilots  use  the  urgency 
level  provided  by  the  master  attention-getters  in  making  their  response 
decisions.  As  seen  in  both  of  the  advanced  systems,  the  response  to  cautions 
is  slower  than  to  the  warnings  even  though  they  were  detected  in  the  same 
time.  This  seems  to  indicate  that  the  pilot  was  making  response  decisions 
based  on  alert  urgency.  Using  the  data  from  System  A,  because  it  is  not 
confounded  with  the  optional  use  of  voice,  the  same  trend  can  be  observed  for 
the  advisory  alerts.  Again,  as  was  found  by  Boucek,  et  al.,  (1980),  if  the 
pilot  has  to  begin  his  response,  i.e.,  looking  at  the  annunciator  panel  to  get 
the  alert  information;  he  will  continue  the  response  regardless  of  the  urgency 
level.  This  can  be  seen  in  the  conventional  system  where  there  is  no  master 
indication  for  advisory  alerts  and  therefore  no  preliminary  urgency 
information;  in  this  case  the  mean  response  times  for  the  advisories  are 
shorter  than  for  the  cautions. 
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Total  perception  of  the  alert  urgency  also  seems  to  be  a  factor  In  a  pilot's 
response.  In  the  advanced  systems  the  warning  and  caution  alerts  receive 
essentially  the  same  treatment  by  the  alerting  system  (master  light,  master 
aural  and  visual  Information  display)  differing  only  In  the  coding  of  these 
components.  On  the  other  hand,  the  conventional  system  treats  warnings  (large 
red  light  and  loud  sound)  quite  differently  from  cautions  (smaller  amber  light 
and  subdued  readout  in  secondary  field  of  view).  The  data  reflects  these 
differences.  There  Is  a  significantly  greater  disparity  in  the  mean  response 
times  between  warnings  and  cautions  for  the  conventional  system  than  for  the 
advanced  systems. 

If  the  flight  crew  includes  a  flight  engineer,  the  results  of  the  study 
indicate  that  his  station  should  contain  the  visual  display  information 
components  of  the  advanced  system.  Making  the  alerting  system  Information 
available  to  the  flight  engineer  in  a  central  location  aids  in  system 
management  and  can  be  used  to  provide  checklists  and  procedural  information. 

Along  with  the  objective  data  obtained  In  the  tests,  pilot  preference  data 
concerning  candidate  concepts  were  also  obtained.  Both  In  the  supplemental 
and  system  validation  tests,  user  reaction  was  very  favorable.  However,  one 
area  which  the  pilots  felt  that  a  change  was  needed  was  the  master  caution 
sound.  Boucek,  et  al ,  (1980)  recommends  the  system  should  take  advantage  of 
pilot's  preconceived  notions  as  to  how  alerts  should  sound.  For  cautions,  a 
constant  or  steady-state  sound  of  mid-range  frequency  was  recommended. 

However,  it  was  also  recommended  that  no  sound  previously  used  on  a  flight 
deck  be  used  for  the  higher  level  alerts  to  avoid  confusion.  When  the 
possible  steady-state  sounds  were  investigated  it  was  discovered  that  they  had 
been  used  or  closely  resembled  a  currently  used  sound.  Therefore,  a  pulsing 
sound  of  short  duration  was  chosen  for  the  caution  alert  (see  Section  5.2.1). 
After  being  exposed  to  this  sound  during  the  training  and  test  flights,  the 
pilots  felt  that  the  sound  was  too  urgent  for  a  caution  level  alert. 

Therefore,  after  reviewing  the  available  data,  (Boucek,  et  al,  1980  and 
Stovner  and  Kelly,  1980)  It  is  recommended  that  a  steady-state  sound  be  used 
for  cautions  and  It  is  further  recommended  that  the  sound  resemble  the  c-chord 
sound  most  commonly  associated  with  caution  level  alerts. 


In  the  final  analysis,  both  candidate  system  concepts  were  validated  as  being 
viable  for  crew  alerting.  Since  the  systems  differed  only  In  their  treatment 
of  the  voice  system.  It  Is  Important  to  consider  the  features  of  each  method 
of  presentation  before  selecting  one  over  the  other. 

Voice  Is  the  recommended  media  of  presentation  when  rapid  response  Is  required. 
Voice  also  permits  the  transfer  of  workload  from  the  visual  to  the  auditory 
channel  (Van  Cott  and  Kincade  1972).  The  system  designer  must  be  aware  of  the 
serial  presentation  requirements  of  the  auditory  channel,  when  selecting  the 
Information  being  presented  aurally.  Boucek,  et  al . ,  (1980)  reported  a 
significant  potential  for  Interference  between  voice  alerts  and  ATC 
communications  when  presented  concurrently.  These  results  would  also  be 
expected  with  other  communication  on  the  flight  deck.  Therefore,  If  voice 
alerts  are  automatically  presented  extreme  care  must  be  used  to  choose  those 
conditions  or  situations  in  which  to  apply  the  voice  alert.  The  condition  or 
situation  must  be  such  that  It  is  highly  unlikely  that  any  other  communication 
on  the  flight  deck  is  more  important  than  informing  the  pilot  of  the  problem. 
This  criteria  Is  met  by  time-critical  warnings.  It  is  therefore  recommended 
that  the  System  A  approach  be  used  for  time-critical  warnings. 

The  type  of  voice  presentation  In  System  B  does  not  depend  on  a  prespecified 
assumption  of  priorities  but  rather  allows  the  pilot  who  is  aware  of  the 
existing  situation  to  select  voice  at  the  appropriate  times.  This  type  of 
design  is  suited  for  transferring  workload  between  the  auditory  and  visual 
channels. 

6.2  TIME-CRITICAL  WARNINGS 

Of  primary  Importance  to  time-critical  warnings  is  the  speed  and  accuracy  of 
the  response.  Therefore,  anything  which  Increases  the  speed  of  alert 
detection  and  response  without  having  an  adverse  effect  on  response  accuracy 
should  be  considered  In  the  system  recommendations. 

The  results  of  this  study  Indicate  that  the  detection  time  measurement  is  very 
sensitive  to  the  presentation  media.  The  fact  that  there  was  no  difference 
found  in  the  mean  detection  time  between  time-critical  and  non-time-critical 
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warnings  Is  not  surprising  since  the  attention-getting  portions  of  both  alerts 
are  the  same.  For  the  time-critical  alerts,  even  though  they  all  had  the  same 
attention-getting  components,  a  significant  difference  was  found  In  the  mean 
detection  times  when  Investigating  the  Interaction  between  the  presentation 
format  and  the  message  Information  type.  If  the  detection  time  were  a  pure 
reaction  to  any  alert  without  regard  to  its  meaning,  no  difference  in  the 
treatment  conditions  would  have  been  expected.  However,  the  method  by  which  a 
“detection"  was  determined  required  the  pilot  to  make  an  overt  action  to 
signify  detection.  This  action  is  a  secondary  task  with  regard  to  responding 
to  the  alert.  It  has  been  shown  that  as  workload  increases,  the  performance 
of  secondary  tasks  changes  (Rolfe,  1971).  Therefore,  it  can  be  postulated 
that  since  the  pilot  begins  processing  information  about  the  alert  as  soon  as 
it  occurs,  the  performance  of  the  secondary  task  of  indicating  an  alert 
detection  may  vary  with  the  amount  of  cognitive  workload  (thinking)  produced 
by  the  alert.  If  this  is  the  case,  then  the  data  indicates  that  the  status 
type  alerts  produce  more  work  for  the  pilot  especially  when  they  are  presented 
graphically. 

The  results  of  response  performance  indicates  that  the  pilot  is  using  urgency 
information  to  influence  his  response.  Even  though  there  was  no  difference  in 
the  detection  time  between  the  time-critical  and  non-time-critical  warnings 
there  was  a  difference  in  mean  response  time. 

The  content  of  the  time-critical  message  is  very  important  in  terms  of  the 
amount  of  time  taken  to  respond.  The  data  indicates  that  guidance  information 
facilitates  the  alert  response.  The  guidance  messages  resulted  in 
consistently  shorter  mean  response  times  when  compared  to  the  status  messages 
regardless  of  the  other  treatment  conditions.  Pilots  preferred  guidance 
messages  for  the  time-critical  situations.  When  guidance  information  is 
presented,  the  data  indicates  that  it  is  most  effective  if  it  is  presented 
graphically.  Care  must  be  used  in  the  development  of  graphic  presentations  to 
insure  that  they  are  designed  properly.  The  pilots,  even  though  they 
performed  better  with  the  graphics,  preferred  the  al phanuneric  presentation. 
They  felt  that  the  guidance  elements  of  the  graphic  display  was  not  given 
enough  emphasis.  If  this  were  the  case,  then  one  might  expect  that  an 
improvement  of  the  graphics  would  result  in  an  even  larger  difference  in 
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performance.  An  Improvement  In  performance  was  also  gained  when  the  graphic 
guidance  alerts  were  placed  In  the  pilot's  primary  field  of  view.  This 
provides  the  pilots  with  an  efficient  presentation  of  the  information  In  the 
location  which  is  closest  to  their  line  of  sight.  Finally,  the  addition  of 
voice  further  improved  the  response  performance. 

In  the  tests  performed  for  the  time-critical  alerts,  the  display  was  structured 
such  that  the  alphanumeric  message  appeared  on  the  left  half  of  the  display 
and  the  graphics  on  the  right.  In  an  English  speaking/writing  society  there 
Is  a  natural  tendency  to  read  from  left  to  right,  top  to  bottom.  It  was 
therefore  expected  that  when  the  alphanumerics  were  combined  with  the  graphics 
the  pilots  would  see  the  alphanumerics  first.  The  rationale  for  this 
placement  was  that  if  the  graphic  presentation  was  being  used  for  the  response 
and  it  was  located  on  the  left  the  pilot  would  not  read  the  alphanumeric 
before  he  responded  and  there  would  be  no  performance  difference  between  the 
graphic  and  combined  formats;  however,  if  the  alphanumerics  were  to  improve 
performance,  putting  them  on  the  left  should  enhance  the  benefit  by  making  it 
more  likely  that  they  will  be  read.  The  results  indicate  that  including  the 
alphanumerics  on  the  left  produces  response  times  that  are  equivalent  to  the 
alphanumeric  message  presented  by  itself  which  are  longer  than  a  graphic 
presentation  of  guidance  information.  This  indicates  that  the  pilots  are 
responding  when  they  have  enough  information  to  permit  a  response  no  matter 
what  the  source  of  the  information.  Therefore,  since  the  data  indicated  that 
graphic  presentations  are  more  effective,  if  alphanumeric  information  is 
presented,  it  should  appear  to  the  right  of  the  graphics. 

Finally,  to  summarize  the  pilot  input  concerning  time-critical  warnings  the 
pilots  felt  that:  an  alert  whi ch  requires  a  time-critical  response  should 
guide  their  actions  with  a  display  which  is  free  of  clutter  and  emphasizes  the 
correct  action;  consideration  should  be  given  to  using  the  EADI  as  the 
time-critical  display  since  it  Is  the  primary  guidance  instrunent;  lastly, 
they  should  be  given  some  kind  of  alert  before  the  situation  reached  the 
time-critical  stage. 


7.0  FUTURE  DIRECTIONS 


The  design  guidelines  contained  in  Volume  II  of  this  report  are  the  product  of 
several  years  of  research  directed  to  the  improvement  and  standardization  of 
flight  deck  alerting  systems.  The  study  has  been  further  directed  toward  the 
display  logic  and  display  presentation  functions  of  a  crew  alerting  system, 
and  operated  within  two  functional  boundaries.  The  investigation  was  bounded 
on  one  hand  by  a  computing  function  which  determined  what  alert  annunciation 
to  the  pilot  was  necessary,  and  on  the  other  by  the  display  function  which 
made  the  pilot  aware  of  the  alert. 

Other  parts  of  the  total  alerting  system  include  the  airplane-specific 
hardware  which  provides  the  multitude  of  information  signals;  the  computer 
which  assimilates  the  signals,  processes  them  in  a  programmed  manner,  and 
decides  what  alerts  are  needed;  and  displays  of  checklist  and  procedural 
information  to  aid  the  pilot  in  handling  the  emergency.  Since  a  systems 
approach  was  used  to  develop  these  guidelines,  it  is  of  interest  to  see  if 
looking  at  a  larger  system  could  reveal  factors  which  might  affect  the  design 
of  the  alerting  system.  A  discussion  of  proposed  tasks  follows. 

7.1  ACCIDENT  IMPLICATIONS  ON  ALERTING  SYSTEM  DESIGN 

It  is  of  interest  investigate  airplane  incident  and  accident  data  to  see  if 
there  are  any  implications  on  alerting  system  design.  Such  an  investigation 
would  consist  of  three  related  tasks;  analysis  of  accident/ incident  data, 
examination  of  the  flight  deck  environment,  and  to  develop  system  functional 
design  concepts  which  could  aid  the  pilot  in  preventing  or  resolving 
emergencies. 

7.1.1  EXAMINATION  AND  ANALYSIS  OF  ACCIDENT/INCIDENT  DATA 

Existing  data  bases  would  be  analyzed  to  identify  factors  that  may  have 
contributed  to  aircraft  accidents  or  Incidents.  The  objective  of  the  review 
would  be  to  evaluate  the  role,  if  any,  that  the  alerting  system  played  in  the 
accidents.  The  review  would  determine  whether  any  dominant  cause(s)  exist 
that  contribute  to  accidents  and  incidents,  and  to  establish  relationships 
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between  the  causal  factors  and  the  alerting  system.  The  ultimate  objective 
would  be  to  ascertain  whether  the  aircraft  alerting  system  was  a  factor  in  the 
accident,  or  if  with  some  modification  could  have  aided  the  pilot  in  the 
emergency. 

The  review  would  also  extract  those  relevant  situations  or  failures  which 
could  create  hazardous  conditions  for  analysis  in  a  later  task. 

7.1.2  EXAMINATION  OF  THE  COCKPIT  ENVIRONMENT 

This  task  would  be  performed  by  observing  commercial  transport  flights  to 
record  flight  deck  operations,  procedures,  and  crewmember  activities.  The 
objective  of  the  task  would  be  to  evaluate  the  environment  in  which  the 
emergencies  occur,  the  flight  deck.  The  investigation  would  include 
evaluation  of: 

•  Alert  Frequency  of  Occurrence 

•  Crewmember  Response(s) 

•  Alerting  System  Operation 

•  Flight  Deck  Operations  and  Activity 

•  Crewmember  Responsibilities  and  Activity 

•  Crewmember  Survey  of  System  Needs 

7.1.3  DEVELOPMENT  OF  NEW  DESIGN  CONCEPTS 

In  this  task  a  limited  number  of  operational  situations  or  aircraft  system 
failures  which  could  possibly  create  hazardous  conditions  would  be  defined. 
These  situations  would  be  used  to  Investigate  the  potential  for  the  alerting 
system  to  monitor  the  aircraft  for  malfunctions  and  failures  and  to  anticipate 
problems  or  expedite  crew  resolution  of  a  problem.  Particular  attention  would 
be  directed  to: 

•  Sensor  Requirements 

•  Time-Critical  Alerts  and  Response  Requirements 

•  Computer  Fault  Diagnosis 

•  Computer  Aided  Decision  Making 
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7.2  EVALUATE  TIME-CRITICAL  DISPLAY  FORMATS 

The  current  study  demonstrated  the  feasibility  of  incorporating  a  special 
display  for  time-critical  alerts.  The  study  also  pointed  out  that  the  manner 
in  which  the  system  information  was  presented  to  the  pilot  was  crucial. 
Improperly  designed  displays  confuse  and  impede  pilot  response,  whereas 
properly  designed  displays  can  elicit  rapid  and  accurate  pilot  response. 
Near-term  needs  for  time-critical  display  of  collision  avoidance,  windshear, 
and  perhaps  active  control  failure(s)  are  anticipated.  Development  of  formats 
for  the  display  of  time-critical  alerts  is  required  and  recommended. 

7.3  INVESTIGATE  INHIBIT  AND  PRIORITIZATION  SCHEMES 

The  current  study  investigated  the  prioritization  of  alerts.  A  group  of 
twenty-one  pilots  was  asked  to  prioritize  a  list  of  given  alerts  into  an  order 
of  relative  urgency/importance.  Two  findings  were  obtained;  effective 
prioritization  should  be  flight  phase  adaptive,  and  there  was  a  high  degree  of 
variability  with  which  the  pilots  prioritized  the  alerts.  Very  little 
analytical  or  empirical  work  has  been  performed  on  bow  alerts  can  be 
prioritized;  further  research  is  necessary  to  develop  this  capability.  A 
similiar  situation  exists  in  the  development  of  inhibit  schemes  for  crew 
alerting.  Both  inhibiting  schemes  and  alert  prioritization  are  strongly 
dependent  on  airplane  design  and  airline  operations.  A  study  broad  enough  to 
encompass  these  two  factors  is  required  and  recommended. 

7.4  DESIGN  AND  EVALUATE  CHECKLISTS 


Methods  for  presenting  checklists  and  procedures  compliment  any  crew  alerting 
system  design.  The  fabrication  of  checklists  and  procedures  is  necessary  for 
each  airplane  design,  and  these  checklists  are  placarded  in  the  flight  deck 
and  in  documents  available  to  the  crew.  Computer  and  display  technology  have 
reached  the  point  where  the  manipulation  and  presentation  of  this  information 
should  be  reviewed  with  the  prospect  of  improving  crew  response  to 
emergencies.  Areas  of  interest  in  such  a  study  include  display  format, 
checklist  accessabil ity ,  and  computer- formulated  checklists. 
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7.5  FLIGHT  VALIDATION  OF  THE  ALERTING  SYSTEM  DESIGN 

Although  the  alerting  system  concepts  were  validated  In  simulation  in  the 
study,  the  design  guidelines  are  essentially  functional  system  characteristics 
and  should  be  further  validated  through  implementation  into  hardware  and 
evaluated  in  flight  tests.  Such  an  evaluation  would  refine  the  guidelines  for 
final  implementation. 
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APPENDIX  A 

TEST  FACILITIES 


A.1  BOEING'S  VISUAL  FLIGHT  SIMULATOR 


The  various  requirements  of  this  study  called  for  an  easily  reconfigurable 
facility  in  which  several  flight  deck  systems  could  be  demonstrated,  tested 
and  evaluated  in  a  realistic  environment.  The  Kent  Visual  Flight  Simulator  at 
the  Flight  Simulation  Center  was  chosen.  Located  in  a  flexible  experimental 
simulation  laboratory,  the  simulator,  called  the  Blue  Cab,  was  modified  to 
represent  a  generic  wide  body  cockpit  configuration  with  working  stations  for 
a  pilot  and  flight  engineer.  The  pilot's  main  instrument  panel  and  the  flight 
engineer's  station  were  designed  to  be  reconfigurable  to  allow  three  different 
alerting  system  display  configurations. 

An  external  visual  workload  was  provided  to  the  pilot  and  flight  engineer 
through  the  forward  windscreens;  computer  controlled  video  and  slides  were 
used  for  takeoff,  target  location,  approach  and  landing.  The  pilot  and  flight 
engineer  were  also  presented  alerting  aural s,  air  traffic  control  commands, 
background  communications,  and  engine  and  aero  sounds. 

The  test  conductor  was  in  visual  and  voice  contact  with  the  pilot  and  flight 
engineer  throughout  the  tests  from  his  console.  This  console  enabled  the  test 
conductor  to  interface  directly  with  the  main  computer  and  control  all  audio 
and  video  parameters.  Figure  A. 0-1  depicts  the  layout  of  the  simulation 
center. 

A.1.1  COCKPIT  SIMULATOR 

The  Cab  had  room  only  for  a  conventional  main  instrument  panel,  standard 
center  console,  and  seats  for  the  pilot  and  copilot.  To  this  cab  a  58  inch 
extension  was  attached.  This  provided  the  necessary  space  for  the  flight 
engineer  station  and  oculometer  camera  mounts,  see  Figures  A. 1.1-1  and  A. 1.1-2. 
The  entire  cab  structure  was  mounted  on  a  hydraulic  platform.  The  cab  was 
situated  towards  the  front  {projection  screen  side)  of  the  lowered  platform. 
This  placed  the  pilot's  eyes  in  an  optimal  relationship  with  respect  to  the 
hemispherical  projection  screen. 


PROJECTOR 


SIDE  VIEW 

Figure  A.1. 1-1.  Side  View  of  Blue  Cab 
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A.1.2  PILOT'S  INSTRUMENTATION 


The  existing  main  forward  instrument  panels,  supporting  framework,  and 
glareshield  were  replaced.  The  replacement  panels  and  glareshield  were 
designed  to  represent  a  generic  wide  body  commercial  aircraft.  The  framework 
and  panels  were  patterned  to  permit  easy  reconfiguration  of  Instruments. 

Figure  A. 1.2-1  shows  the  front  panel  used  in  the  Time-Critical  test 
configuration. 

The  center  console  was  modified  to  accept  two  keyboard  units  for  pilot  data 
entry.  An  overhead  console  containing  three  working,  lighted  engine  fire 
handles  was  also  added.  The  pilot's  instrumentation  consisted  of  raster  scan 
CRT's,  standard  electro-mechanical  instruments,  annunciators  and  switches.  A 
(9  inch)  color  Hitachi  CRT  was  used  for  the  EADI.  It  was  driven  from  a  Boeing 
built  graphics  generator.  Two  (5  inch)  color  Hitachi  CRT's  were  used  to 
present  advanced  and  time-critical  alerting  messages  and  graphics.  Both  CRT's 
were  driven  by  one  Lexidata  model  3400  color  graphics  generator.  A  (9  inch) 
black  and  white  CRT  was  used  to  display  engine  Instrument  information.  It  was 
driven  by  a  Boeing  built  bar  graphics  generator.  The  video  generators  were 
all  driven  directly  from  the  host  computer. 

The  servo  and  synchro  motors  of  the  electro-mechanical  instruments  were  driven 
from  a  local  controller.  Digital  information  from  the  host  computer  was  fed 
to  the  controller.  The  controller  then  passed  it  through  digital  to  analog 
and  digital  to  syncro  converters.  The  lighted  annunciators  were  driven  by 
discrete  output  cards.  Discrete  Input  cards  sampled  the  switches  when 
requested  by  the  host  computer. 

A.1.3  FLIGHT  ENGINEER  STATION 

A  partially  active  Flight  Engineer  (F.E. )  station  was  installed  on  the  right 
rear  side  wall  of  the  cab,  see  Figure  A. 1.3-1.  Six  active  panels  were 
designed  and  installed  in  addition  to  a  general  alerting  system  annunciator 
panel,  situation  response  keyboard,  and  numeric  entry  keyboard.  All 
annunciators,  switches  and  dials  were  controlled  in  the  same  manner  as  those 
on  the  pilot's  panels.  These  etched  panels  were  back-lit  to  simulate  normal 
instrumentation. 
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A.  1.4  TEST  CONDUCTOR  CONSOLE 


Located  next  to  the  cab,  on  the  main  floor  of  the  simulation  room,  the  test 
conductor  console  provided  a  good  view  of  the  projection  screen,  the  pilot 
station,  and  the  audio  and  video  equipment.  Through  a  terminal  on  the 
console,  the  test  conductor  controlled  the  simulation  host  computer  and  varied 
simulation  parameters.  An  intercom  system  permitted  communication  with  the 
host  computer  room,  the  model  room,  and  the  cab.  The  console  layout  is  shown 
in  Figure  A. 1.4-1. 

The  audio  equipment  rack  was  located  next  to  the  console.  Most  of  the  audio 
system  controls  were  remoted  to  the  console,  and  the  close  proximity  of  the 
equipment  rack  afforded  easy  visual  verification  of  actions  taken.  A  2-80 
microcomputer,  used  to  control  the  voice  alerting  system,  was  mounted  below 
the  console.  A  second  terminal,  located  to  the  right  of  the  console, 
controlled  the  Z-80  and  displayed  voice  system  information. 

A  19  inch  black  and  white  CRT  mounted  in  the  oculometer  rack  displayed  the 
pilots  instrument  panel.  The  CRT  was  positioned  so  the  test  conductor  could 
easily  monitor  the  pilot's  actions  from  the  console. 

A.1.5  HOST  COMPUTER  AND  SIMULATION  EQUIPMENT  I/O 

All  simulation  equipment,  including  the  flight  instruments,  were  controlled  by 
the  simulation  host  computer  through  a  chaining  I/O  controller  (or  chain 
controller).  The  chain  controller  on  instruction  from  the  host  computer 
passed  data  to  selected  instruments  (or  hardware)  or  retrieved  data  from  the 
simulator.  The  chain  controller  also  Interfaced  with  the  test  conductor's 
console  and  the  model  room.  The  chain  controller  cycled  at  a  rate  of  2.5  to 
10  milliseconds.  Maximum  usage  brought  It  down  to  10  milliseconds  per  cycle. 
Therefore,  the  maximum  delta  between  a  pilot’s  or  F.E.’s  action  and  the 
notation  of  that  action  was  approximately  one  one-hundredth  of  a  second. 
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Figure  A.  1.4 ■  1.  Test  Conductor  Console  Layout 


A.1.6  SIMULATION  HOST  COMPUTER 


The  simulation  host  computer  was  comprised  of  three  Varian  V75  computers 
operating  in  parallel.  A  nine-track  magnetic  tape  system  was  used  to  record 
pilot  responses,  flight  parameters,  and  flight  data. 

A.  1.7  VISUAL  SYSTEM 

The  outside  visual  scene  was  projected  on  a  thirty  foot  diameter  hemispherical 
screen.  The  outside  visual  scene  consisted  of  either  still  slide  projections 
or  a  moving  scene.  The  moving  scenes  were  provided  by  a  dual  closed-circuit 
servo  camera  system.  Each  camera  viewed  a  three-dimensional  model  board  in 
the  model  room.  The  cameras  were  mounted  on  carriages  that  traveled  over  the 
length  of  the  terrain  boards.  One  board  was  used  for  take-off,  final  approach 
and  landing.  Initial  approach  was  provided  by  the  second  camera  and  another 
terrain  board.  The  scene  was  projected  from  a  black  and  white  projector  that 
was  mounted  on  top  of  the  cab,  directly  over  the  pilot's  head.  The  host 
computer  coordinated  the  camera/carriage  servos  with  the  pilot's  control  of 
the  airplane  resulting  in  a  realistic  outside  visual  scene.  Figure  A. 1.7-1 
shows  the  layout  of  the  video  system. 

For  target  locating  and  reporting  workload  tasks  three  slide  projectors 
displayed  pseudo  targets  on  the  screen.  The  slide  projectors  were  advanced  by 
the  host  computer  but  turned  on  and  off  by  the  test  conductor  at  the  console. 

A.1.8  AUDIO  SYSTEM 

The  audio  equipment  was  mounted  in  a  double  width,  six  foot  high  equipment 
rack,  see  Figure  A. 1.8-1.  Figure  A. 1.8-2  shows  how  the  equipment  was 
interconnected.  Number  one  reel-to-reel  tape  player  provided  engine  and  aero 
noise.  Number  two  reel-to-reel  tape  player  provided  the  alerting  voice 
recordings.  The  control  of  this  player  by  a  Z-80  micro-computer  will  be 
discussed  below.  Cassette  recorder  number  one  recorded  the  pilot's  and  F.E.'s 
voices.  Cassette  number  two  provided  ATC  messages.  Cassette  number  three 
provided  ATC  background.  An  equalizer  was  used  to  shape  the  engine  and  aero 
noises  to  represent  a  heavy  jet  aircraft. 
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Figure  A.  1. 7- 1.  Video  System  Layout 


Figure  A.  1.8  1.  Audio  Equipment  Rack  Layout 


The  host  computer  had  direct  control  of  thirteen  discretely  triggerable  aural 
alerts.  Eleven  of  the  alerts  were  generated  by  a  production  B-747  aural 
warning  box,  i.e.,  buzzer,  horn,  intermittent  horn,  wailer,  hi-chime, 
lo-chime,  hi-lo-chime  fast  repeat  cl  acker,  bell  and  c-chord.  The  other  two 
aural  alerts  (i.e.,  European  siren  and  "electronic  owl")  were  produced  by 
specially  designed  and  built  circuits. 

Audio  mixers  were  used  to  mix  the  various  sounds  for  presentation  to  the  pilot 
and  flight  engineer.  They  also  combined  the  various  sounds  with  the  pilot’s 
and  flight  engineer's  voices  for  the  test  conductor. 

The  voice  alert  tape  player  had  recordings  of  thirty-seven  different  voice 
alerts  with  each  alert  repeated  twenty  times.  Each  alert  series  of  twenty  was 
separated  from  the  adjacent  alerts  by  a  physically  transparent  space  on  the 
tape.  The  Z-80  microcomputer  controlled  the  tape  player  by  sensing  the 
passage  of  the  clear  tape  spaces  and  kept  track  of  which  message  was  at  the 
playing  head.  The  host  computer  would  send  a  code  to  the  Z-80  specifying  a 
particular  voice  message,  allow  about  45  seconds  for  the  Z-80  to  locate  the 
message,  command  the  Z-80  to  play  the  message,  and  then  command  it  to  stop. 

A.1.9  OCULOMETER  AND  VIDEO  MONITORING  SYSTEM 

The  Flight  Dynamics  Laboratory  at  Wright-Patterson  Air  Force  Base  loaned 
Boeing  their  Honeywell  Mark  V  Oculometer  to  be  used  in  this  study  to  gather 
data  on  the  pilot's  scan  patterns.  The  Air  Force  also  provided  engineering 
support  to  assist  in  the  set-up,  calibration,  and  operation  of  the  oculometer 
system. 

Unfortunately,  due  to  several  hardware  problems  and  a  constrained  schedule, 
the  oculometer  was  not  operational  for  use  in  the  study.  The  oculometer' s  low 
light  level  video  scene  system  (camera  and  19"  monitor)  did  work  well,  though, 
so  the  scene  camera  system,  looking  over  pilot's  right  shoulder,  was  utilized 
to  give  the  test  conductor  a  good  view  of  the  pilot's  actions,  and  the  entire 
instrument  panel.  This  helped  the  test  conductor  monitor  the  simulated 
flights.  The  output  of  the  camera  was  also  connected  to  a  video  tape 
recorder,  providing  the  test  conductor  with  the  capability  of  recording  an 
entire  flight  if  desired. 
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A.2  DIGITAL  EQUIPMENT  TECHNOLOGY  ANALYSIS  CENTER  (DETAC) 

A.2.1  GENERAL  DESCRIPTION 

DETAC  is  a  technology  investigation  facility  used  for  the  purpose  of 
conducting  studies  and  providing  hands-on  experience  for  engineers  assigned  to 
tasks  associated  with  digital  equipment.  This  facility  was  established  to 
fulfill  a  requirement  to  upgrade  the  existing  electronic  system  evaluation 
capabilities,  particularly  in  the  area  of  aircraft  digital  systems,  inclusive 
of  flight  control  computers  and  advanced  display  concepts.  The  facility  as 
well  as  the  cockpit  fixture  are  illustrated  in  Figure  A.2. 1-1. 

Aircraft  that  utilize  several  different  types  of  digital  computer  systems 
require  careful  study  of  software  structure,  allocation  of  hardware/software 
function,  redundancy  management,  etc.,  to  obtain  proper  reliability  and  safety 
of  flight.  The  DETAC  system  permits  engineers  to  investigate  problem  areas, 
conduct  real-time  simulations,  and  monitor  the  design  and  integrity  of  vendor 
production  hardware. 

The  DETAC  facility  has  the  following  operational  features: 

•  Central  digital  computer  with  a  real-time  operating  system,  96K  words  of 
memory  with  memory  mapping,  floating-point  hardware,  and  cache  memory. 

•  FORTRAN  IV  software  package. 

•  Interactive  graphics  unit  with  FORTRAN  level  software  package. 

•  Three  satellite  digital  computers  with  32K  words  of  memory,  512-word 
writable  control  store,  and  floating-point  firmware  package. 

•  High  speed,  party-line  communications  link,  500K  words/sec. 

•  Cockpit  simulation  apparatus  and  displays. 

( 
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The  DETAC  facility  Is  used  to  support  advanced  commercial  and  military  studies 
In  digital  flight  controls.  Integrated  cockpit  technology,  aircraft  mutiplex 
systems,  and  advanced  military  tactical  displays.  Specific  types  of  digital 
avionics  Investigations  Include  system  architecture  and  stability  studies, 
digital  autopilot  evaluation  and  mechanization  studies;  higher-order  language 
applications;  hardware,  software,  firmware  tradeoffs;  display  format  studies; 
and  software  reliability  and  certifiabil ity  studies. 

DETAC  has  five  major  elements:  Central  computer  and  peripherals,  satellite 
computer  and  peripherals.  Interactive  graphics,  general  input/output  (I/O) 
hardware,  and  cockpit  fixtures.  There  basic  functional  elements  and 
associated  equipment  are  shown  In  Figure  A. 2. 1-1. 

The  Sperry  Univac  V76  minicomputer  is  general  purpose  and  micro-programmable. 

A  cache  enhances  memory  access  for  faster  operation.  Three  Sperry  Univac  V76 
minicomputers  provide  computations  and  simulation  support  for  the  central 
computer.  Peripheral  support  equipment  includes  a  Dec-Writer  111  terminal, 
card  reader,  magnetic  tape,  cassette  tape.  Century  Data  CDS-114  disc,  Infotron 
Vistar/GT  al phanuneric  display  terminal,  and  a  Varian  Statos-31 
printer/pl otter. 

Interactive  computer  programs  define  wind  shears,  turbulence,  ILS 
characteristics,  aircraft  initial  conditions,  and  fl ight-control -system 
parameters.  Simulations  of  advanced  flight-guidance  system  utilize  the  unique 
capability  of  DETAC  to  operate  several  computers  asynchronously  in  parallel. 
Multiple  computers  simulate  redundant  avionics  systems,  while  other  computers 
simulate  the  head-up  visual  scene  cockpit  displays.  Data-reduction  programs 
plot  selected  simulation  parameters  on  a  Vector  General  display  and  the  Statos 
electrostatic  plotter. 

This  fixture  consists  of  a  wooden  mockup  of  a  three  man  wide  body  cockpit  with 
crew  seats,  control  mechanisms,  CRT  displays,  instrument  panel  and  supporting 
structure  Interfaced  to  the  Varian  computers.  The  DETAC  cockpit  can  be  easily 
reconfigured  to  permit  the  study  of  advanced  cockpit  concepts.  The  instrument 
panel  consists  of  several  color  and  black-and-white  CRT's.  A  microprocessor 
provides  a  flexible  Interface  between  the  cockpit  controls  and  the  satellite 
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computers.  Head-up  and  external  visual  displays  are  produced  by  an  Advent 
television  projection  system. 

A.2.2  FACILITY  MODIFICATIONS  FOR  AIRCRAFT  ALERTING  SYSTEM  STANDARDIZATION 
STUDY 

Several  facility  modifications  were  made  prior  to  Phase  I  testing.  Figure 
A. 2.2-1  represents  a  schematic  diagram  of  the  hardware  configuration  that  was 
used.  An  experiments  control  station  was  added  as  well  as  a  microprocessor 
interface  unit  for  activation  of  warning  system  devices  in  the  cockpit.  A 
prototype  synthetic  voice  warning  system  was  Installed  along  with  audio 
equipment  for  ambient  noise  simulation  and  ATC  communications.  In  addition,  a 
GENESCO  color  graphics  system  was  added  to  drive  the  central  caution  and 
warning  display  unit.  Figure  A. 2. 2-2  illustrates  the  specific  modifications 
made  in  the  DETAC  cockpit.  As  can  be  seen,  a  modular  overhead  panel  was 
installed  to  accommodate  lighted  switches  to  be  used  for  simulated  fault 
identification  and  correction.  Three  sets  of  speakers  were  provided  for  the 
transmission  of  simulated  ATC  communications.  The  side  panel  speakers  were 
used  to  introduce  alert  messages  and  the  floor  mounted  speakers  were  employed 
to  simulate  ambient  cockpit  noise.  A  number  of  instrument  panel  configuration 
changes  were  also  made.  These  modifications  are  shown  in  Figure  A. 2. 2-3. 

Master  warning  and  caution  lights  were  added  to  the  glareshield  as  well  as  a 
Central  Aural  Warning  System  cancel  switch  which  functions  to  cancel  and  reset 
any  on-going  auditory  alerts.  A  control  display  unit  with  its  associated 
control  keys  Is  located  at  the  center  of  the  instrument  panel  while  a  hard 
wired  annunciator  matrix  was  installed  in  the  Captain's  primary  field  of  view 
for  required  warnings  and  system  status  information  in  an  operational  system, 
the  annunciator  matrix  would  serve  as  a  back  up  device  to  be  used  during 
control  display  unit  failure.  It  is  presented  here  for  demonstration  purposes 
and  was  not  a  part  of  the  experimental  tests.  Figure  A. 2. 2-4  shows  the 
pilot's  position  relative  to  the  two  side  panel  speakers.  As  can  be  seen,  his 
head  is  positioned  between  these  two  speakers  while  the  ATC  speaker  is 
positioned  directly  above  his  head.  The  pilot's  eye  position  corresponds 
roughly  to  the  design  eye  reference  point.  Control  of  the  aircraft  is 
exercised  by  means  of  a  side  stick  controller  as  Illustrated  In  Figure 
A. 2.2-5.  The  overhead  response  panel  used  to  acknowledge  fault  messages  as 
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Figure  A.2.2-2.  Modification  Made  in  DETAC  Cockpit  for  Tests  III  and  IV 


CENTRAL  AURAL  WARNING  SYSTEM  CANCEL  SWITCH 


Figure  A.2.2-3.  Instrument  Pane!  Configuration 


i 


Figure  A. 2.2-4.  Pilot  Position  Relative  to  Side-Mounted  Speakers 


Figure  A.  2. 2-5.  Sidestick  Controller,  Overhead  Response  Panel,  and 
Glareshield-Mounted  Microphone 


well  as  the  glareshield  mounted  microphone  used  to  record  the  pilot's  verbal 
responses  are  also  visible  in  Figure  A. 2.2-5.  The  external  visual  scene  was 
generated  on  a  computer  graphics  terminal  and  presented  directly  in  front  of 
the  pilot  on  an  ADVENT  projection  screen  at  a  distance  of  approximately  15 
feet.  The  projection  screen  can  be  seen  In  relation  to  the  cockpit  fixture  in 
Figure  A. 2. 2-6.  Head-Up  Display  (HUD)  symbology  was  used  by  the  pilot  for 
visual  guidance  during  simulated  approaches.  The  pilot's  task  was  to  maintain 
the  aircraft  symbol  centered  over  the  command  symbol.  The  difficulty  of  this 
two  axis  tracking  task  could  be  modified  by  introducing  various  levels  of 
turbulence.  A  representation  of  the  HUD  symbology  as  used  in  Phase  I  can  be 
seen  in  Figure  A. 2. 2-7. 

As  can  be  seen  in  Figure  A. 2. 2-1,  control  of  the  experiments  was  maintained 
from  a  remote  location  within  the  DETAC  facility.  With  this  configuration  it 
was  possible  to  initiate  each  test  trial  and  introduce  the  appropriate  alert 
messages  without  having  to  enter  the  cockpit  fixture.  Video  taping  equipment 
was  installed  to  record  the  pilot's  movements  and  verbal  responses  for 
subsequent  analysis. 
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APPENDIX  B 


PICTORIAL  REPRESENTATIONS  OF  INTERACTIVE 
CAPABILITIES  OF  ADVANCED  ALERTING 
SYSTEM 


w 


FIGURE  B?  CURSOR  INDEXED  AT  FLAP  LIMIT  INOP” 


H  2 


FIGURE  B4.  MESSAGE  ("FLAP  LIMIT  INOP  '  REVERTS  TO  MEMORY 
Not<;  deferred  item  imlir-i'm  0  1  m  tn.v  t  <  uiht  t.or  n*-r  t 


FIGURE  B5.  DEPRESS  "MODE"  KEY  TO  ENTER  MEMORY  MODE- 


FIGURE  B6.  DEPRESS  "MODF"  KEY  AGAIN  TO  RF  Vli:T  RAIK  TO  AttRTMODF 


FIGURE  B7  SYSTEM  REVERTS  BACK  TO  ALERT  MODE' 


Em-  lit  BH  DEPRESS  LIME  ADVANCE  KEY  TO  MOVE  CURSOR  UP  TO 
CABIN  A I  TITIJDf 
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FIGURE  (59  CURSOR  INDEXED  AT  CABIN  ALTITUDE” 
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FIGURE  BIO.  DEPRESS  "CKL"  KEY  TO  BRING  UP  EMERGENCY  PROCEDURE 
CHECKLIST  FOR  "CABIN  ALTITUDE"  (RAPID  DECOMPRESSION) 


FIGURE  B11 .  EMERGENCY  PROCEDURE  CHECKLIST  DISPLAYED  FOR  "CABIN 
ALTITUDE"  (RAPID  DECOMPRESSION) 
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FIGURE  B12.  SYSTEM  OVERFLOW  CONDITION  WITH  3  WARNINGS,  2  CAUTIONS. 


AND  1  ADVISORY  ON  PAGE  2,  AS  INDICATED  BY  COLORED  BOXES  AT 
THE  BOTTOM  OF  SCREEN.  The  deferred  item  indicator  (Ml)  in  lower  right 
corner  indicates  that  one  message  has  been  stored  in  memory.  The  number  (1) 
in  lower  left  corner  indicates  that  page  one  of  the  alert  mode  is  being  viewed. 
Note  that  warning  messages  are  red,  caution  —  amber,  and  advisories  —  blue. 


FIGURE  B1 3.  DEPRESS  "PAGE''  KEY  TO  TRANSITION  FROM  PAGE  1  TO  PAGE  2 


WHERE  THE  ADDITIONAL  MESSAGES  CAN  BE  VIEWED  (3  WARNINGS, 
2  CAUTIONS,  AND  1  ADVISORY) 
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FIGURE  B14  SYSTEM  REVERTS  TO  PAGE  2.  Note  page  indicator  in  lower  right  corner. 
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FIGURE  B15.  BOX  AROUND  ALERT  ("DC  BUS  1  OFF")  INDICATES  THAT  IT  IS  THE 
MOST  RECENT  MESSAGE'  This  is  method  1  for  annunciating  the  most 
recent  alert. 


FIGURE  B16.  ASTERISK  (*)  NEXT  TO  MESSAGE  (FIRE  ENGINE  3)  INDICATES  THAT 
IT  IS  THE  MOST  RECENT  MESSAGE.  This  is  method  2  for  annunciating 
the  most  recent  alert. 
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APPENDIX  C 


FAA 

AIRCRAFT  ALERTING  SYSTEMS  STANDARDIZATION  STUDY 
SYSTEM  COMPONENT/LOGIC  EVALUATION 

DEBRIEFING  SUMMARY 
RESULTS 


ORGANIZATIONS  REPRESENTED 


Organization  Number  of  Pilots 


Douglas  Aircraft  Company  5 
Boeing  Commercial  Airplane  Company  2 
Lockheed  California  Company  4 
Continental  Airlines  6 
Western  Airlines  8 
Total  Number  of  Pilots  Participating:  25 


AIRCPAFT  TYPES  REPRESENTED  (VEHICLE  OF  MOST  RECENT  EXPERIENCE) 
Aircraft  Types  Number  of  Pilots 


DC-9  2 

DC-10  6 

B727  12 

B747  1 

L1011  4 


MEAN  NUMBER  OF  FLIGHT  HOURS:  11,319 
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A.  MASTER  ALERT  (WARNING  AND  CAUTION) 


1.  In  the  context  of  an  advanced  alerting  system  the  master  light  will 
serve  not  only  as  a  visual  alert,  but  also  as  an  alerting  medium 
(not  center  screen  message)  cancellation  device.  In  your  judgment, 
which  of  the  following  alternatives  will  be  most  appropriate  for  use 
as  visual  alert? 


N 

11  Split  Master  Light  (Warning  and  caution  lights  on  the  same 
switch) 

14  Separate  Master  Lights  (Warning  and  caution  lights  on  separate 
switches) 


Can  you  think  of  any  Instances  where  separate  master  lights  would  be 
necessary?  10  Yes  15  No 


Comments:  a) 
-  b) 

c) 

d) 

d) 

e) 

f) 

9) 

h) 

1) 

j) 


Dual  warnings.  (2) 

Familiar  with  separate  master  light  so  prefer  them.  (1) 
Different  order.  (1) 

Fire  warning,  engine  fuel  shut  off  switches  and  fire  handle 
(separate  master).  (1) 

Fire  warning  should  be  delineated.  (1) 

Same  system  as  DC-10  with  engine  fires.  (1) 

Dual  multiple  emergencies  or  abnormal s  (situation  where  amber 
condition  creates  an  emergency  condition). 

In  cancelling  a  caution  message,  a  warning  message  could  be 
cancelled  without  knowing  it.  (1) 

Definite  master  light  for  different  light.  (1) 
when  direct  sun  light  is  on  master  lights.  (1) 

MEL /reminder  items.  (1) 


B.  MASTER  ALERT  (ADVISORY) 


1.  Which  of  the  following  medium  do  you  feel  would  be  most  appropriate 
as  master  alert  for  advisory  level  information? 


N 

4  Master  light 
0  Discrete  tone 
0  Voice 

3  Flashing  box  around 
most  recent  message 
1  Master  light  and  tone 


N 

2  Master  light  and  voice 
12  Master  light  and  flashing  box 
2  Tone  and  flashing  box 
1  Other 


2.  Advisory  information  display.  Which  of  the  following  do  you  feel  Is 
the  most  appropriate  means  by  which  advisory  level  Information 
should  be  displayed? 


N 

0  Voice 

23  Alpha  numeric  display 

2  Voice  plus  alpha  numeric  dlpslay 

0  Other 
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C.  STORE/RECALL 

1.  Two  methods  are  presently  available  for  storing  messages  and 

recalling  them  from  memory.  A  total  store/ recall  capability  would 
be  desirable  as  It  would  reduce  the  Interactive  time  requirements  of 
the  system.  Can  you  foresee  any  situations  where  a  selective 
store/recall  capability  would  be  desirable  or  necessary? 


Comments:  a) 

b) 

c) 

d) 

e) 


f) 

g) 

hi 


When  phase  of  flight  or  problems  are  of  different  order  of 
importance.  (1) 

Should  be  capable  of  both,  so  low  priority  items  can  be 
delayed.  (1) 

Multiple  sector  flights  where  continuation  using  redundant 
system  to  increase  departure  reliability.  (1) 

Should  be  capable  of  both  (avoid  unwanted  messages  but  also 
provide  a  full  message  capability.  (2) 

When  problem  display  has  been  attended  to  with  all  procedures-- 
but  problem  can't  be  solved  it  should  be  moved  to  "store"  or 
reduced  to  "advisory  only"  because  no  other  action  can  be 
taken .  ( 1 ) 

MEL  items/ low  priority  items.  (1) 

Should  have  priority  built  into  it  on  importance  of  certain 
critical  systems  of  the  airplane.  (1) 

More  options  available  to  tailor  unit  to  respective  airlines 
checklist  procedures.  (1) 


2.  Overall,  which  feature  would  you  prefer? 


N 

8  Total  store/recall 
3  Selective  store/recall 

14  Combination  of  total  and  selective  store/recall 
0  Other 

a)  Automatically  goes  into  store  after  all  "checklist"  items  have 
been  completed.  (1) 

b)  Flexability.  (1) 

c)  Total  store  recall  should  be  minimum  required  but  some 
situation  select  recall  could  be  used.  (1) 
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D.  PROCEDURAL/CHECKLIST  INFORMATION 


How  effective  was  the  presentation  of  procedural  information  that  was 
demonstrated? 

N 

7  Excellent  -  No  changes  recommended 

16  Good  -  Minor  changes  beneficial 

2  Fair  -  Minor  changes  recommended 

0  Poor  -  Major  changes  recommended 

0  Very  Poor  -  Major  changes  necessary 

Please  describe  any  changes  or  recommendations  you  feel  should  be  made 
regarding  procedural  information. 

Comments:  a)  CRT  was  not  readable.  (1) 

b)  Checklist  automatically  displayed  for  all  warnings.  (1) 

c)  Two  warnings  occurred  at  the  same  time--highest  priority 
displayed  first.  (1) 

d)  Item  being  accomplished  could  be  flashed  or  boxed.  (1) 

e)  Current  checklists  utilize  a  logic  flow  pattern  where  choices 
are  offered  which  could  be  incorporated.  (1) 

f)  Procedural  development  of  checklist  response  by  either  asterisk 
movement  or  line  cancellation.  (1) 

g)  System  of  checking  or  noting  each  item  on  list  as  it  is 
accomplished.  (1) 

h)  System  would  have  to  be  developed  to  avoid  aircraft  action 
without  thought--if  it  is  required  for  flight-performance 
consideration.  (1) 

i)  Critical  items  should  be  isolated  from  non-critical  data.  (1) 

j)  Have  display  uncluttered  of  all  but  critical  items.  (1) 

k)  Procedure  items  should  be  color  coded  the  same  as  information. 
(1) 

l)  Some  method  to  cross  through  each  step  as  completed.  (1) 

m)  Immediate  action  items  should  be  in  bold  face  presentation.  (1) 

nj  Priority  could  be  beened  to  be  established  on  items  that  could 

or  would  selectively  be  stored--as  immediate  and  secondary 
actions  are  completed  itsm  should  be  automatically  stored.  (1) 

o)  Slightly  larger  screen  would  be  easier  to  read.  (1) 

p)  The  checklist  should  provide  interaction  where  possible  when 
the  steps  of  the  check  are  completed.  (1) 

q)  The  checklist  information  will  disappear  when  check  off  or 
completed--? 

r)  Color  should  be  employed  using  the  same  priority  method  as 
master  light— include  cruise  data  callup  and  system  trouble 
shooting  procedures.  (1) 

s)  Color  dots  could  be  used  to  emphasize  items  completed  or  items 

to  be  completed  as  well .  (1) 
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E.  RANK  THE  FOLLOWING  OPTIONS  FOR  ATTENUATION/CANCELLATION  OF  AURAL  ALERTS 

Number  of  #1  Rankings 

2  a"J  Automatic  vol ume  reduction  after  a  fixed  number  of  repetitions. 

0  b)  Manual  vol  line  reduction. 

4  c)  Automatic  cancellation  after  a  fixed  number  of  repetitions. 

19  d)  Manual  cancellation  with  automatic  cancellation  after  problem 
corrected. 

Consider  the  option  that  you  rated  most  h.jhly.  Briefly  explain  why. 

1.  Automatic  volume  reduction  after  a  fixed  number  of  repetitions. 

Comments:  a)  Manual  cancellation  of  item  is  done  without  conscious  thought. 

(1) 

b)  Only  the  most  critical  warnings  should  be  aural  and  then  only 
for  fixed  number  of  repetitions  due  to  the  nunerous  things 
going  on  in  the  cockpit.  (1) 


2.  Automatic  cancellation  after  a  fixed  number  of  repetitions. 


Comments:  a)  Reduce  the  number  of  action  required  by  aircrew.  During  an 

emergency  also  extraneous  sounds  in  the  cockpit  are  distracting 
once  emergency  has  been  identified.  (1) 

b)  Gives  the  crew  the  information  they  are  required  to  perform  an 
action  in  order  to  answer  a  computer.  (1) 

c)  Least  distracting  for  pilot  flying  the  aircraft--should  also 
include  a  automatic  volume  reduction  after  first  alert.  (1) 


3.  Manual  cancellation  with  automatic  cancellation  after  problem 

corrected. 

Comments:  a)  Aural  warnings  can  be  annoying  if  continues--we  only  need  a 

attention  getter--should  be  able  to  be  silenced  manually.  (1) 

b)  Make  it  possible  to  cancel  aural  alert  to  avoid  distraction 
once  problem  acknowledged--! f  corrected  itself  as  with  power 
shift  then  would  want  auto  cancellation.  (1) 

c)  Terminate  voice  as  soon  as  possible  because  it  interferes  with 
ATC  communication.  (1) 

d)  Any  alert  should  require  positive  action  by  the  pilot  to  cancel 
problem  signal.  (1) 

e)  Places  pilot  in  the  decision-making  process  to  think  about  the 
problem  with  time  left  up  to  him.  (3) 

f)  Gives  flexibility  of  manual  with  automatic  feature.  (1) 

g)  Consider  inhibition  warnings  during  critical  take  off  area.  (1) 

h)  Enables  retention  as  long  as  desired  and  maybe  cancelled  at 
discretion.  (2) 
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1)  Once  Attention  Is  drawn  to  the  Item  and  acknowledged,  the  aural 
only  provides  a  distraction.  (2) 

j)  Pilot  should  cancel  warning,  because  auto  cancel  could  cause 
signal  to  be  missed.  (1) 

k)  Requires  a  positive  action  thereby  assuring  corrective  action 
of  the  fault  (after  corrected  It  should  be  dropped).  (1) 

l)  Because  he's  use  to  It.  (2) 

m)  During  many  phases  of  flight  master  warning  and  caution  alerts 
would  have  to  wait  for  flight  maneuvering  to  be  completed.  (1) 

n)  ATC  can  override  many  alerts--distraction  of  this  type  in 
critical  phase  or  flight  situation  must  be  manually  cancellable 
under  other  conditions  auto  cancellation  is  desirable. 

o)  With  automatic  cancellation  with  alot  of  messages  one  could 
think  the  problem  was  solved  if  he  no  longer  heard  alerts  after 
respond  to  problem  annoying  to  have  verbal  non-  cancelling 
message  continue.  (1) 

F.  VISUAL  MESSAGE  SYNTAX 

MIL-STD-411D  recomnends  the  following  format  for  verbal  alert  messages: 

General  Heading  Specific  Subsystem  Nature  of  Emergency 

_  _ Or  Location _ _ 

Engine  Number  1  Hot 

Which  format  do  you  feel  would  be  most  appropriate  for  visual  messages? 

N 

20  a)  General  Heading  -  Subsystem/Location  -  Nature  of  Emergency/ 
Condition 

3  b)  Nature  of  Emergency/Condition  -  General  Heading  - 
Subsystem/Location 
2  c)  Other 

Please  explain  the  rationale  behind  your  selection. 

Comments:  a)  Data  presented  needs  to  be  in  logical  sequence  for  the  pilot. 

(1) 

b)  Nature  of  the  problem  (first),  where  the  problem  is  (second), 
and  delay  the  third.  (1) 

c)  Identify  a  problem,  the  system,  the  location,  and  logical 
progression  of  what  to  do.  (1) 

d)  Both  depending  upon  the  item— fire  Engine  No.  1,  oil  pressure 
low.  (1) 

e)  Identify  item  priority  to  declaring  the  nature  of  the 
emergency.  (1) 

f)  General /subsystem/location,  nature/condition.  (1) 

g)  Emergency  situation-nature  of  emergency— then  identify  correct 
engine  or  whatever.  (1) 

h)  Nature  of  emergency  most  important.  (1) 

1)  Delete/?  of  emergency  (Engine  No.  1)  associated  instruments 
would  indicate  problem.  (1) 


G.  VOICE  ALERT 


It  has  been  suggested  that  the  words  "warning"  and  "caution"  may  be  more 
effective  than  auditory  tones  as  precursors  to  corresponding  alert 
messages.  Which  of  these  alternatives  do  you  feel  would  be  most 
appropriate? 

N 

15  Words  better  than  tones  as  master  alert 
9  Tones  better  than  words  as  master  alert 
1  No  difference 


Please  explain  the  rationale  behind  your  selection. 
1.  Words  better  than  tone  as  attenson. 


Comments:  a) 

b) 

c) 

d) 

e) 

f) 

g) 

h) 
1) 

j) 

k) 

n 

m) 


With  CRTs  fewer  categories  of  information  would  have  to  be 
remembered.  (1) 

Excessive  number  of  warning  sounds  presently  incorporated.  (4) 
Proliferation  of  tones  becomes  a  confusion  factor.  (1) 

Word  warning  could  increase  set  response  action  especially  when 
ti red .  ( 1 ) 

Need  to  incorporate  state-of-the-arts  technology,  but  also 
accommodate  progressive  flight  crew  transition.  (1) 

Noise  level  in  the  flight  station  could  mask  the  aural  level. 
Don't  have  to  memorize  tones--under  stress  they  (tones)  can  be 
confused.  (1) 

With  all  the  tones  already  present,  word  would  be  better.  (3) 
Should  have  a  tone  associated  with  a  word.  (1) 

No  translation  required,  therefore  no  misunderstanding.  (1) 
Quicker  recognition.  (1) 

Takes  the  guess  work  out.  (1) 

Word  serves  a  double  purpose  of  attention  and  acknowledgement 
of  problem.  (1) 


2.  Tones  better  than  words  as  master  alert. 


Comments:  a) 

b) 

c) 

d) 

e) 

f) 


Tendency  to  "filter"  voice  to  hear  what  is  needed  at  the  time. 
(1) 

Easier  recognized  if  the  number  of  tones  are  kept  to  a 
minimum--there  is  too  much  conversation  in  cockpit  already.  (I) 
Distinct  tone  or  sound  may  be  more  recognizable  under 
interfering  conditions.  (1) 

Words  O.K.  for  low  level  problems,  but  for  attention  grabbing 
tasks  a  "noise"  is  superior.  (1) 

Cockpit  is  continually  full  of  radio  chatter,  so  appropriate 
tone  warning  would  be  most  useful.  (1) 

Tones  are  adequate--but  words  would  be  better--but  don't  need 
them .  ( 1 ) 
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H.  SEQUENCING  OF  MULTIPLE  VERBAL  ALERTS 

|  Rank  the  following  options  for  sequencing  of  multiple  verbal  alerts.  If 

}.'  you  can  suggest  a  viable  alternative,  please  describe  it  briefly  in  the 

i  space  provided  and  rank  it  accordingly. 

Number  of  #1  Rankings 

S  al  Prioritize  messages  and  annunciate  only  the  most  severe 
problem.  Annunciate  subsequent  message( s)  only  after  the 
previous  one  has  been  corrected  or  somehow  accommodated. 

Ob)  Introduce  each  message  for  a  fixed  number  of  repetitions, 

cancel  it  and  introduce  the  next  one.  This  would  also  require 
■  prioritization  within  gross  alert  category. 

14  c)  Introduce  the  message  "multiple  alerts"  and  direct  flight  crew 
attention  to  central  display  unit  for  specific  fault  messages. 

3  d)  Other 

i 

4.  Other 

i 

Comments:  a)  Introduce  all  new  messages  and  prioritize  with  a  flashing  box, 
which  would  move  on  by  priority. 

b)  Boxing  will  draw  the  operators  attention  to  the  most  serious 
and  still  allow  total  view  of  the  problem.  (1) 

c)  Voice  alerts  cannot  be  utilized  on  multiple  alerts--can  only  do 
one  at  a  time. 

I.  GENERAL  COMMENTS 

1.  Chevron  should  advance  with  line  as  messages  occur.  (3) 

2.  Possible  indent  caution  and  advisories.  (1) 

.  3.  Procedure  might  come  on  automatically.  (1) 

4.  Be  sure  you  don't  add  to  the  crew  workload  with  a  bunch  of  new 
buttons,  etc.  (1) 

t 

5:  There  are  too  many  aural  alerts  (tones)  in  conventional  cockpits. 

(4) 

> 

6.  There  is  an  excessive  number  of  nuisance  alerts  and  false  warnings 
in  the  cockpit  today.  (1) 

7.  The  alerting  system  needs  to  be  centralized  in  the  cockpit.  (1) 

8.  It  might  be  appropriate  to  indent  cautions  and  advisories  so  a  loss 
would  not  cause  a  loss  of  the  pilot  ability  to  differentiate  time 
essential  from  low  priority  alerts.  (1) 

9.  With  a  prioritization  system,  procedural  information  might  come  on 
automatically  for  the  most  severe  alert.  (7) 
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10.  Display  major  problem  and  subsidiary  problems  indented  below  it.  (1) 


11.  Simplicity  and  reliability  are  the  most  important  characteristics  of 
the  alerting  system.  {1) 

12.  New  or  most  recent  message  could  be  annunciated  by  increasing  its 
intensity  on  the  visual  display  unit. 

13.  A  back  up  CRT  could  be  used  for  normal  operation  checklists  as  well 
as  emergency  procedures.  (1) 

14.  Lack  of  pilot  agreement  on  alert  prioritization  and  inhibit  logic 
will  serve  to  highlight  the  need  for  a  good  prioritization  and 
inhibit  system.  If  system  designers  have  trouble  prioritization 
alerts  when  gathered  around  a  meeting  table,  it  may  be  unrealistic 
to  expect  pilots  to  prioritize  and  act  on  alerts  during  a  multiple 
failure  situation,  characterized  by  high  levels  of  workload,  stress 
and  confusion.  (1) 
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APPENDIX  D 


FAA 

ALERTING  SYSTEM  STANDARDIZATION  STUDY 
SYSTEM  COMPONENT  EVALUATION 
DEBRIEFING  QUESTIONNAIRE 
RESULTS 


ORGANIZATION  REPRESENTED 


Organization  _N 

Douglas  Aircraft  Company  4 

Boeing  Commercial  Airplane  Company  2 

Lockheed  California  Company  4 

Continental  Airlines  4 

Western  Airlines  _3 

Total  Number  of  Pilots  Responding  13 

AIRCRAFT  TYPE  MOST  RECENTLY  EXPERIENCED  WITH 
Aircraft  Type 

DC-9  4 

DC-10  1 

B727  8 

L-1011  4 


MEAN  NUMBER  OF  FLIGHT  HOURS:  13,019 


FAA 

AIRCRAFT  ALERTING  SYSTEM  STANDARDIZATION  STUDY 
SYSTEM  COMPONENT  EVALUATION 


DEBRIEFING  QUESTIONNAIRE 


NAME: 

DATE: 

ORGANIZATION  REPRESENTED: 

AIRCRAFT  TYPE  MOST  RECENTLY 

EXPERIENCED 

WITH: 

DC-10 

B -707 

B-737 

L-1011 

OC-9 

B-727 

B -747 

OTHER 

NUMBER  OF  FLIGHT  HOURS: 

MAIL  COMPLETED  QUESTIONNAIRE  TO: 

David  Po-Chedley 
Mail  Code  35-36 
Douglas  Aircraft  Co. 

3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 


VISUAL  ALERTING  SYSTEM 
DEBRIEFING  QUESTIONNAIRE 


Candidate  Concepts  To  Be  Evaluated: 

1.  Chronology  within  color  { CC ) :  New  message  appears  at  the  bottom  of 
its  alert  group. 

2.  Reverse  Chronology  within  color  (RCC):  New  message  appears  at  the 
top  of  its  alert  group. 

3.  Priority  (P):  New  message  appears  at  the  pre- determined  location 
within  its  alert  group  depending  on  its  priority  level. 

4.  Basic  Chronology  (BC):  Host  recent  message  appears  at  the  top  of 
the  list,  regardless  of  color  or  specific  priority  level. 


Scoring  Options: 

1.  Excellent  -  No  changes  recommended. 

2.  Good  -  Hinor  changes  beneficial. 

3.  Fair  -  Minor  changes  recommended. 

4.  Poor  -  Major  changes  recommended. 

5.  Very  Poor  -  Major  changes  necessary. 


Please  Circle  The  Appropriate  Number 

A.  OVERFLOW  LOGIC 

1.  How  good  was  the  overall  logic  of  the  display  format? 


cc 

1 

2 

3 

4 

5 

Mean 

Rating 

2.66 

RCC 

1 

2 

3 

4 

5 

2.33 

P 

1 

2 

3 

4 

5 

2.58 

BC 

1 

2 

3 

4 

5 

3.26 

Rate  the  importance  of  the  above  characteristic/feature 
relative  to  its  effect  on  safe  aircraft  operations 
during  failure  or  emergency  conditions. 

Mean  Importance  Rating:  1.62 

12  3  45 

Very  Somewhat  Neither  Important  Somewhat  Very 

Important  Important  Nor  Unimportant  Unimportant  Unimportant 

2.  How  well  would  you  be  able  to  evaluate  the  aircraft  status? 


CC 

1 

2 

3 

4 

5 

Mean 

Rating 

2.26 

RCC 

1 

2 

3 

4 

5 

1.86 

P 

1 

2 

3 

4 

5 

2.41 

BC 

1 

2 

3 

4 

5 

2.93 

Rate  the  importance  of  the  above  characteristic/ 
feature  relative  to  its  effect  on  safe  aircraft 
operations  during  failure  or  emergency  conditions. 

Mean  Importance  Rating:  1.57 

1  2  3  4  5 

Very  Somewhat  Neither  Important  Somewhat  Very 

Important  Important  Nor  Unimportant  Unimportant  Unimportant 
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3.  How  good  is  each  concept  at  helping  to  avoid  confusion 
about  alert  priority  level? 


1 

Very 

Important 


CC 

RCC 

P 

BC 


1 


2 

2 

2 

2 


4 

4 

4 


Rate  the  importance  of  the  above  characteristic/ 
feature  relative  to  its  effect  on  safe  aircraft 
operations  during  failure  or  emergency  conditions. 


Mean  Importance  Rating:  1.50 


Somewhat 

Important 


Neither  Important 
Nor  Unimportant 


Somewhat 

Unimportant 


Mean 

Rating 

2.66 

2.33 

2.53 

3.27 


5 

Very 

Unimportant 


4.  How  good  is  the  probability  of  avoiding  errors  with  the  concept? 


Mean 

Rating 


CC 

1 

2 

3 

4 

5 

2.67 

RCC 

1 

2 

3 

4 

5 

2.20 

P 

1 

2 

3 

4 

5 

2.76 

BC 

1 

2 

3 

4 

5 

3.26 

Rate  the  importance  of  the  above  characteristic/ 
feature  relative  to  its  effect  on  safe  aircraft 
operations  during  failure  or  emergency  conditions. 

Mean  Importance  Rating:  1.50 

1  2  3  4  5 

Very  Somewhat  Neither  Important  Somewhat  Very 

Important  Important  Nor  Unimportant  Unimportant  Unimportant 


How  well  can 
of  alerts? 

you  identify 

the  chronological 

order 

Mean 

Rating 

CC 

1 

2 

3 

4 

5 

2.40 

RCC 

1 

2 

3 

4 

5 

2.07 

P 

1 

2 

3 

4 

5 

3.27 

BC 

1 

2 

3 

4 

5 

1.86 

1 

Very 

Important 


Rate  the  importance  of  the  above  characteristic/ 
feature  relative  to  its  effect  on  safe  aircraft 
operations  during  failure  or  emergency  conditions. 


Mean  Importance  Rating:  2.38 


Somewhat 

Important 


Neither  Important 
Nor  Unimportant 


4 

Somewhat 

Unimportant 


5 

Very 

Unimportant 


Questions 


Importance  rating  scale: 

1.  Very  important 

2.  Somewhat  important 

3.  Neither  important  nor 
unimportant 

4.  Somewhat  unimportant 

5.  Very  unimportant 


Display  format: 

□  Chronology  within  color 

•  Reverse  chronology 
within  color 

O  Priority 

♦  Basic  chronology 


Figure  D-1.  Mean  Pilot  Ratings  for  Overflow  Logic  Options  (With  95%  Confidence  Limits) 
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How  well  can  you 
N  identify  the  chronological 
®  order  of  alerts? 


Please  Circle  The  Appropriate  Number 


Please  answer  the  questions  in  Sections  B,  C  and  D  using  the  five-point  rating 
scale  (Excellent  to  Very  Poor)  shown  on  Page  1. 

B.  BASIC  DISPLAY  CHARACTERISTICS  Mean 

Rating 

1.  Character  Spacing  12345  1 .71 

2.  Word  Spacing  12345  1.76 

3.  Line  Spacing  12345  2.00 

4.  Stroke  Width  12345  2.12 


5.  Contrast  With  Background 

a)  Red  Messages  12345  1.64 

b)  Yellow  Messages  12345  1.70 

c)  Blue  Messages  12345  2.05 

d)  White  Peripheral  Information  1  2  3  4  5  2.11 

C.  PERIPHERAL  INFORMATION 


1.  Rate  the  following  characteristics  of  the  mode  indicator 
(Alert/ Memory)  on  the  display  screen. 


a)  How  well  does  it  indicate 
system  mode? 


Mean 

Ratine 


1  2  3  4  5 


b)  How  effectively  is  it  located 
on  the  screen? 


1  2  3  4  5 


c)  How  effective  is  the  color 
(White)  in  helping  to 
differentiate  it  from 

the  alert  messages?  12345  2.35 

d)  Do  you  feel  there  is  a  better  way  of  presenting  this 
information?  If  so,  please  explain. 


2.  New  Message  Indicator  (Asterisk) 


a)  How  well  does  it  aid  in 
detecting  the  most  recent 
alert  for  the  first  time? 

b)  How  effective  is  the  flash  12345  2.35 

rate  in  drawing  your  attention 

to  a  new  message? 

3.  New  Message  Indicator  (Colored  Box  Around  Message) 

a)  How  well  does  it  aid  in  1  2  3  4  5  1.41 

detecting  the  most  recent 

alert  for  the  first  time? 

b)  How  effective  is  the  flash  12345  1.35 

rate  in  drawing  your  attention 

to  a  new  message? 

c)  How  effective  would  it  be  12345  3.00 

if  used  to  replace  the 

master  lights  in  the  pilots 
primary  field  of  view? 

d)  Do  you  feel  that  there  is  a  better  way  to  present  this 
information?  If  so  please  explain. 


Mean 


4.  Page  Number  Indicator 

a)  Size  1  2  3  4  5 

b)  Location  12345 

c)  Contrast  With  Alert  Messages  12345 

d)  Clarity  Of  Meaning  12345 


2.47 


2.88 
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e)  Do  you  feel  there  is  a  better  way  of  presenting  this  information? 


5.  Cursor  (Line  Advance  Indicator) 

Mean 

a)  Size  1  2  3  4  5 

b)  Shape  1  2  3  4  5  2.06 

c)  Dynamics  Relative  to  Key  Input  12345  2.13 

d)  Contrast  With  Alert  Messages  12345  2.00 

e)  Do  you  feel  there  is  a  better  way  of  presenting  this  information? 


6.  Colored  Boxes  Used  To  Indicate  The  Presence  Of  An  Overflow  Condition 
(Additional  Messages  On  Page  2  And/Or  3). 

Mean 

a )  Si  ze  1  2  3  4  5 

b)  Shape  1  2  3  4  5  1.94 

c)  Location  12345  2.00 

d)  Ease  Of  Differentiation 
Between  Warning,  Caution 
And  Advisory  Message 

Indicators  12345  2.00 


e)  Effectiveness  Of  Flash  Rate 
Used  To  Indicate  The  Presence 

Of  New  Message  On  Page  2  Or  3  1  2  3  4  5  1.88 

f)  Color  Contrast  Between 
Colored  Boxes  And  Messages 

Within  Them  12345  2.00 

g)  Ability  To  Avoid  Confusion 
Regarding  The  Number  Of 
Messages  For  Each  Alert 
Level  Being  Stored  On 

Pages  2  And  3  12345  2.70 

h)  Do  you  feel  there  is  a  better  way  of  presenting  this  information? 


7.  Deferred  Item  Indicator  (Memory  Symbol) 

a)  Size  1 

b)  Shape  1 

c)  Location  1 

d)  Ability  To  Differentiate 
Between  Messages  Being 
Stored  In  Memory  And 
Those  Being  Stored  On 

Pages  2  And  3  1 

e)  Ability  To  Avoid  Confusion 

Regarding  Number  Of  Messages 
Being  Stored  In  Memory  1 

f)  Clarity  Of  Meaning  1 


Mean 

Rating 

2.28 
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g)  Do  you  fuel  there  is  a  better  way  of  presenting  this  information 


8.  Function  Keys  (Located  To  The  Left 

Of  ' 

rhe  Display  ! 

Screen) 

Mean 

a) 

Si  ze 

1 

2 

3 

4 

5 

Rating 

1.87 

b) 

Shape 

1 

2 

3 

4 

5 

1.80 

c) 

Location 

1  -  Mode  Key 

1 

2 

3 

4 

5 

2.00 

2  -  Check! ist  Key 

1 

2 

3 

4 

5 

1.93 

3  -  Line  Advance  Key 

1 

2 

3 

4 

5 

1.86 

4  -  Store  Key 

1 

2 

3 

4 

5 

1.86 

5  -  Page  Key 

1 

2 

3 

4 

5 

1.79 

d) 

Contrast  Between  Legends 

And  Backgrounds 

1 

2 

3 

4 

5 

2.07 

e) 

How  Weil  Do  The  Display 
Dynamics  Correspond  To 

Key  Inputs? 

1 

2 

3 

4 

5 

2.14 

9.  Line  Keys  (Located  To  The  Right  Of 

The 

Display 

Screen) 

Mean 

a) 

Si  ze 

1 

2 

3 

4 

5 

Rating 

T754 

b) 

Shape 

1 

2 

3 

4 

5 

2.53 

c) 

Location 

1 

2 

3 

4 

5 

2.53 

d) 

Contrast  Between  Legends 

And  Backgrounds 

1 

2 

3 

4 

5 

2.46 

e) 

How  Well  Do  Display  Dynamics 
Correspond  To  Key  Inputs? 

1 

2 

3 

4 

5 

2.58 

I 


f)  Do  you  feel  there  is  a  better  way  of  presenting  this  information 


D.  USE  OF  COLOR  TO  DIFFERENTIATE  BETWEEN  WARNING,  CAUTION  AND 
ADVISORY  MESSAGES 

Mean 

Rating 

1.  How  Well  Can  You  Distinguish 

Between  Alert  Priority  Level?  12345  1.31 

2.  How  Effective  Is  Color  As  An 
Aid  In  The  Evaluation  Of 

Aircraft  Status?  12345  1.18 

3.  How  Well  Does  Color  Help  Avoid 
The  Possibility  of  Confusing 

Alert  Priority  Level?  1  2  3  4  5  1.25 

4.  If  You  Were  To  Suggest  A  More 
Effective  Color  Coding  Scheme 
As  An  Alternative  To  Red, 

Yellow,  And  Blue,  What  Colors 
Would  You  Recommend? 


APPENDIX  E 


FAA 

ALERTING  SYSTEM  STANDARDIZATION  STUDY 
PRIORITIZATION  AND  INHIBIT  QUESTIONNAIRE 
RESULTS 


El 


ORGANIZATION  REPRESENTED 


Organization 

Douglas  Aircraft  Company 
Boeing  Commercial  Airplane  Company 
Lockheed  California  Company 
Continental  Airlines 
Western  Airlines 

Total  Number  of  Pilots  Responding 


Number  of  Pilots 
7 
2 

3 

4 
_5 
21 


AIRCRAFT  TYPES  REPRESENTED  (VEHICLE  OF  MOST  RECENT  EXPERIENCE) 


Ai rcraft  Type 
DC-9 
DC-10 
B727 
b  747 
L - 101 1 

MEAN  NUMBER  OF  FLIGHT  HOURS:  11,669 


Number  of  Pilots 

3 

5 

9 

1 

3 
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UNCLASSIFIED  FAA/RD-81/36/1  NL 


Means  and  95%  confidence  limits  for  the  16  alerts 
used  in  the  prioritization  questionnaire: 

1.  Anti  Skid  Left  Inboard  Failure 

2.  APU  Fire 

3.  Left  Fuel  Pump  Valve  Open 

4.  Battery  Bus  Off 

5.  Generator  Off 

6.  Galley  Overheat 

7.  APU  Generator  Off 

8.  Cabin  Pressure  Relief  Valve  Open 

9.  Wing  Anti  Ice  Disagree 

10.  Air  Condition  Pack 

11.  Duct  Avionic  Compartment  Overheat 

12.  Engine  Fire 

13.  Left  Emergency  AC  Bus  Off 

14.  GPWS 

15.  Manifold  Failure  (Pneumatic) 


16.  Cabin  Altitude 


MEAN  RATING  5  MEAN  RATING 


re  E-1.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts— Antiskid  L  Inbd  Fail 


FLIGHT  PHASE 

A  /  B 


110 
100 
90 
80 
70 
60 
SO 
40 
X 
20 

Figure  E-2.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts- APU  Fire 


MEAN  RATING  _  MEAN  RATING 


figure  E-3.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts— L  Fuel  Dump 


Valve  Open 


Figure  E-4.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts-Battery  Bus  Off 


E-5 


MEAN  RATING  MEAN  RATING 


Figure  E-5.  Meant  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts-Gen  Off 


Figure  E-6.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts-Gallery  Overheat 

E4 


MEAN  RATING  >)  MEAN  RATING 


E-7 


MEAN  RATING  5  MEAN  RATING 


Figure  E- 10.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts - 
Air  Cond  Pack  Off 
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FLIGHT  PHASE 


Figure  E-11.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts 
Duct  Avionic  Comp  Overheat 


FLIGHT  PHASE 


WARNING 


CAUTION 
9 


A/.  JSF/  £ 


*5?  /3&V& 

w  /  ^Va^oW  ^Vy  / 


WARNING 


CAUTION 
9 


Figure  f •  12.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts— Eng  Fire 


MEAN  RATING  MEAN  RATING 


Figure  E-15.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts— 
Manifold  Fail  (Pneu) 


Figure  E- 16.  Means  and  Confidence  Limits  for  Pilot  Ratings  of  Selected  Alerts— Cabin  Alt 
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Means  and  95%  confidence  limits  for  the  eight  flight 
phases  used  In  the  prioritization  questionnaire: 


Flight  Phase 

1.  Taxi 

2.  Initial  Take  Off 

3.  Final  Take  Off 

4.  Initial  Climb 

5.  Climb/Cruise/Descent 

6.  Initial  Approach 

7.  Final  Approach 

8.  Landing  Rollout 


Definition 

0  kts  -  30  kts  to  400  feet 
Vj  -  30  kts  to  400  feet 

1,500  feet  to  200  feet 
200  feet  to  Touchdown 
Touchdown  to  Taxi 
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Figure  E-17.  Means  and  95%  Confidence  Limits  for  Pitot  Ratings  of  Selected  Caution- 
Level  Alerts-  Taxi 
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Figure  E- 18.  Means  and  95%  Confidence  Limits  for  Pilot  Ratings  of  Selected  Caution- 
Level  Alerts— Initial  Takeoff 
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Figure  E- 19.  Means  and  95%  Confidence  Limits  for  Pilot  Ratings  of  Selected  Caution 
Level  Alerts- Final  Takeoff 
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Figure  E-20.  Means  and  95%  Confidence  Limits  for  Pilot  Ratings  of  Selected  Caution- 
Level  Alerts— Initial  Climb 


MEAN  RATING  MEAN  RATING 


Figure  E-2k  Means  and  95%  Confidence  Limits  for  Pilot  Ratings  of  Selected 
Caution-Level  Alerts— Climb  /Cruise  /Descent 


Figure  E-22.  Means  and  95%  Confidence  Limits  for  Pilot  Ratings  of  Selected 
Caution-Level  Alerts— Initial  Approach 
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MEAN  RATING 


Figure  E-23.  Means  and  95%  Confidence  Limits  for  Pilot  Ratings  of  Selected 
Caution-Level  Alerts— Final  Approach 


Figure  E-24.  Means  and  95 %  Confidence  Limits  for  Pilot  Ratings  of  Selected 
Caution-Level  Alerts— Landing  Rollout 
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ALERTS 


1.  ANTISKID  L  IN9D  FAIL  I  19  I  19 


2.  APU  FIRE  10  25 


3.  L  FUEL  DUMP  VALVE  OPEN  24  29 


4.  BATTERY  BUS  OFF  15 


5.  GEN  OFF  10 


6.  GALLEY  OVERHEAT  19 


7.  APU  GEN  OFF  25 


8.  CABIN  PRESS  RELIEF  VALVE  OPEN  28  I  47 


9.  WING  ANTI-ICE  DISAGREE  38  47 


0.  AIR  COND  PACK  OFF  24 


11.  DUCT  AVIONIC  COMP  OVERHEAT  15  35 


12.  ENG  FIRE  14  19 


13.  LEMER  AC  BUS  OFF  25  35 


14.GPWS  32  42 


15.  MANIFOLD  FAIL  (PNEU)  23 


16.  CABIN  ALT  45  I  55 


47 
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25 
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33 
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APPENDIX  F 


PILOT  TRAINING  CHECKLIST 


P-i 


Pilot  Briefing  Checklist 


INTRODUCTION 

1.  Background 

a.  This  is  an  FAA  program  that  develops  a  systematic 
approach  to  crew  alerting. 

b.  The  program  is  a  three-phase  effort;  phases  1  and  2 
are  completed. 

c.  The  first  phase  evaluated  elements  of  the  alerting  system 
and  developed  candidate  system  concepts. 

d.  The  second  phase  provided  the  design  of  tests  that 
evaluated  candidate  systems. 

You  will  participate  in  these  tests. 

2.  Phase  3  objectives 

a.  Resolve  problems  and  include  results  in  test 

b.  Validate  the  advanced  pilots  design 

c.  Validate  the  advanced  flight  engineer  design 

d.  Evaluate  time-critical  presentation  media 

e.  Provide  guidelines  for  designers  of  alerting  systems 

FLIGHT  TASK 

1.  Active  displays 

a.  EADI 

b.  HSI/DME 

c.  Airspeed 

d.  Altimeter 

e.  Vertical  speed 

f.  Clock 

g.  Alert  display(s) 

h.  Engine  instrument 

i.  Flaps 

j.  12  key 

2.  Active  controls 

a.  Wheel  and  column 

b.  Rudder  and  toe  brake 

c.  Speed  brake 

d.  Flaps 

e.  Gear 

f.  Fire  handle 

g.  Response  switches 

h.  12  key 

i.  Throttles 


Pilot  Briefing  Checklist  ( Concluded ) 
~iT  FLIGHT  TASK  (Continued) 

3.  Flightpath 

a.  Takeoff 

b.  Climb 

c.  Cruise 

d.  Descent-cloud  layer 

e.  Land 

f.  Turns 

g.  Autothrottle 

h.  Windshear 

i.  Updates 

4.  ATC 

a.  Flightpath  direction 

b.  Traffic  annunciation 

III.  CREW  ALERTING 

1.  Advanced  system  displays 

a.  Information 

b.  Master  visual 

c.  Master  aural 

d.  Voice  alerts 

e.  Time  critical 

f.  EADI  change 

2.  Conventional  system  display 

a.  Distributed  alerts 

b.  Annunciator  panel 

c.  Discrete  tones 

3.  Alert  response 

a.  Flight  management  responses 

b.  System  management  responses 

4.  Review  alerts  and  responses 

IV.  TRAINING  FLIGHTS 

1.  Airplane  familiarization  flight 

a.  Review  handling 

b.  Introduce  ATC  guidance 

c.  Familiarization  with  flight  plan 

2.  Advanced  system  familiarization 

a.  Review  possible  alerts 

b.  Review  responses 

3.  Conventional  system  familiarization 

a.  Review  alerts 

b.  Review  responses 
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APPENDIX  G 


SYSTEM  VALIDATION 
AND 

TIME-CRITICAL  TEST 
DEBRIEFING  QUESTIONNAIRE 


o-i 


Observer  No. 


Name: 


Data: 


Phone: 


Age  __X_=4fi£ 


Number  of  year*  flying:  X  »  274? 


Approximate  number  of  flight  hour*:  X  *  13.600 


In  the  space  below,  identify  the  types  of  aircraft  you  have  flown .  Put  a  1  above  the  aircraft  type  you  have 
flown  most  recently,  a  2  above  the  next,  and  so  on. 


(B-707)  (B-727)  (B-737)  (B  747)  (DC-9)  (DC-10)  (1-1011) 


(DC-6) 


(  >  (  ) 

Other 


0-2 


DEBRIEFING  QUESTIONNAIRE 


OBSERVER 


DATE 


A  SYSTEM  COMPARISON 


Advanced 

About 

Conventional 

system 

equal 

system 

1 .  Which  system  provided 

□ 

□ 

□ 

easier  problem  idr-  'ification? 

86% 

7.5% 

7.6% 

2.  Which  system  was  most 

□ 

□ 

□ 

effect**;-  n  gettiij  your 
attention* 

92.5% 

7.6% 

3.  Which  system  would  you 

l8t% 

□ 

□ 

prefer  for  the  crew  alerting 
system-' 


4.  Which  visual  system  was 
easier  to  use? 

92?5% 

□ 

□ 

5.  Which  tone  system  was 

85% 

□ 

□ 

easier  to  use? 

7.5% 

7.6% 

Tone- 

About 

Tone 

voice 

equal 

only 

6.  Which  aural  alerting  mode 

□ 

□ 

□ 

was  most  effective  in 
getting  your  attention? 

61.5% 

31% 

7.6% 

7.  Which  mode  would  you 

Q 

□ 

□ 

prefer  in  the  cockpit? 

68% 

23.5% 

7.6% 

Too 

About 

Too 

low 

equal 

high 

8.  Evaluate  aural-alerts 

□ 

□ 

8% 

loudness  with  respect 
to  aircraft  noise. 

77% 

Comments: 
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8.  ADVANCED  SYSTEM  COMPONENT  EVALUATION 


INSTRUCTIONS:  FOR  EACH  OF  THE  QUESTIONS  BELOW.  ASSIGN 
A  SCALE  VALUE  FROM  1  TO  S  BY  CHECKING 
THE  APPROPRIATE  BOX. 


Scale  value*: 

1  -  Unacceptable— major  change*  necetsary 

2  -  Poor  -major  change*  recommended 

3  ■  Fair-minor  change*  recommended 

4  -  Good— minor  change*  beneficial 

5  =  Excellent -no  change*  recommended 


Rating 

1 

2 

3 

4 

5 

1 .  How  appropriate  was  the 
location  of  the  visual 
information  display? 

□ 

□ 

□ 

7.5% 

Rx 

□ 

15.5% 

2.  Rate  the  following 

characteristics  of  the  visual 
information  display. 

a.  Character  size 

□ 

□ 

15£% 

S« 

□ 

15.5% 

b  Color 

□ 

□ 

1?5X 

Sx 

i?.5% 

c.  Brightness 

□ 

□ 

iSbX 

Sx 

1^6% 

d.  Character  separation 

□ 

□ 

?,x 

Sx 

8% 

e.  Message  content 

□ 

□ 

8x 

Sx 

8% 

f.  Flashing  box 

□ 

□ 

8x 

Sx 

8% 

3.  Evaluate  the  tones  selected 
for  this  study. 

a.  Warning 

□ 

a* 

I5?5X 

s% 

8% 

b.  Caution 

□ 

7?5% 

Sx 

Sx 

□ 

c.  Advisory 

□ 

□ 

8x 

Sx 

□ 

31% 

Rating 


1 

2 

3 

4 

5 

4.  Rate  the  following 
characteristics  of  the 
tones. 

a.  Attention-getting  value 

□ 

□ 

□ 

□ 

31% 

□ 

89% 

b.  Information  content 

□ 

□ 

7.5% 

□ 

23% 

□ 

54% 

□ 

15% 

c.  Potential  for  disruption 

□ 

□ 

S» 

8* 

fi% 

d.  Number  of  tones 

□ 

□ 

7.5% 

A* 

S* 

31% 

5  Evaluate  the  ease  of  use 
of  the  voice  system. 

□ 

□ 

□ 

7.5% 

□ 

54% 

□ 

38% 

6.  Evaluate  the  voice  component 
with  respect  to  the  confusion 

□ 

□ 

□ 

□ 

38% 

□ 

62% 

of  the  alert*  with  other 
communication. 

7.  Rate  the  voice  component 
on  the  following 
characteristic*. 


a.  Attention  getting 

□ 

□ 

□ 

□ 

46% 

□ 

54% 

b.  Intelligibility 

□ 

□ 

□ 

15% 

□ 

31% 

□ 

54% 

c.  Message  content 

□ 

□ 

7.5% 

□ 

7.5% 

□ 

23% 

□ 

62% 

d.  Loudnesss 

□ 

□ 

fla 

38% 

□ 

46% 

e.  Repetition  rate 

□ 

7£% 

8* 

3% 

□ 

38% 

f.  Voice  type 

□ 

□ 

A» 

3* 

□ 

54% 

Comments: 


S-S 


C.  TIME -CRITICAL  ALERTS 


Center 

panel 

i| 

Pilots 

panel 

1.  At  which  location  did  you  see 
the  alert  the  fastest? 

□ 

8. 

8s 

2.  Which  location  do  you  prefer 
for  time-critical  alerts? 

3.  Which  alert  format  is  easier 
for  you  to  use? 

□ 

Guidance 

£% 

8% 

About 

equal 

8* 

8s 

Status 

□ 

4.  Which  format  would  promote 
the  quickest  response  for 
a  time-critical  alert? 

8* 

8* 

□ 

16% 

5.  Which  format  do  you  prefer 
for  the  time-critical  alerting 
system? 

77% 

□ 

23% 

□ 

Graphic  Alpha 

Both 

No 

difference 

6.  Which  type  of  presentation 
is  easiest  to  use? 

D  s* 

□ 

7.  Which  type  of  presentation 
will  promote  the  fastest 
response? 

□  □ 

60% 

8* 

?» 

8.  Which  type  of  presentation 
will  result  in  the  fewest 
errors? 

□  □ 

64% 

3% 

8% 

9.  Which  type  of  presentaiton  do 
you  prefer  for  the  time- 

ft 

□S 

O 

8 * 

□ 

critical  alerting  system? 


Comments: 


SYSTEM  FEATURE  IDENTIFICATION 


INSTRUCTIONS.  Considering  all  the  alerting  concepts  that  have  been  used  during  your  test  flight. 

please  identify  below  five  features  of  the  alerting  system  that  you  liked  best: 


1 .  Single  tone  for  warning,  caution,  and  advisory  68% 

2.  Voice  on  cautions  50% 

3.  Central  location  for  information  85% 

4.  Volume  of  aural  alerts  31% 

5.  Use  of  voice  31% 

6.  Distinct  color  for  urgency  levels  23% 

Five  features  that  you  liked  least : 

1.  Time-critical  graphics  54% 

2.  Caution  tone  too  urgent  38% 

3. 

4. 

5. 


Five  changes  that  you  would  make  (if  any) : 

1 .  Use  EADI  for  time-critical  69% 

2. 

3. 

4. 

5. 
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